Empirical Legal Studies

Digital Archive of Blackmum Papers

Great news for Supreme Court researchers.

With support from the National Science Foundation, Jeff Segal, Harold Spaeth, and Lee Epstein have just rolled out the  Digital Archive of the Papers of Harry A. Blackmun. In its present form, the Archive houses .pdfs of the docket sheets and preliminary (pool) memoranda from Blackmun's years on the Rehnquist Court (1986-1993). We hope its contents will interest a wide range of users, from those looking for the votes (and clerk recommendations) in specific cases to those hoping to study trends in the Court's case-selection and decision-making processes.  The Archive's homepage is here. To go directly to the Archive, use this link: http://epstein.law.northwestern.edu/research/BlackmunArchive/

Creating the Archive required Segal, Spaeth, and Epstein to work with hundreds of thousands of digital images.  Therefore, as users spot problems, please send an email to Lee Epstein  ([email protected]).  The researchers will work to quickly correct errors as they become known.  Finally, Jeff, Harold, and Lee realize that the Archive works (and looks) better in some browsers than others.  They are hoping to fix the problem soon.

Science for the Masses(?)

Via Slashdot: an interesting post asks, "Is scientific journalism doomed?":

"(H)ow can you effectively present primary scientific literature to the general public?  Is this even possible?"

...There are a number of assumptions I have been forced to make while analyzing my data, many of which are critical for both my methodology and the development of few of my arguments...How do you explain to someone the relative magnitude of these assumptions?...How, then, do you effectively cover a story laden with valid assumptions, some likely to be correct, many likely to be incorrect?"

As they say, read the whole thing.  For anyone who has ever tried to present empirical work to a non-empirical audience, it's a familiar problem.

Data on Usage Wars Waged Over "Data" and "Datum"

Every since graduate school when a member of my dissertation committee lit into a draft paper of mine I developed a healthy pet peeve about the word "data" (the plural form of the singular "datum" -- or so I thought). Over at the Social Science Statistics blog Mike Kellermann's recent post gigs into the usage wars waged over data and uncovers some ambiguity.

Mike appears to throw his lot behind the "data is a singular because it is increasingly view as a 'mass noun'" camp. For me, however, Strunk & White's classic and time-tested The Elements of Style breaks any usage "ties." Regrettably, even here, while the tie is broken, a hint of change exists. "Data. Like strata, phenomena, and media, data is a plural and is best used with a plural verb. The word, however, is slowly gaining acceptance as a singular."

New FBI Crime Data

The FBI released the 2006 Crime in the United States report today. The 2006 report is here. Previous reports are here.

Today's FBI press release is here.

Advice for Erwin Chemerinsky

Per the announcement last week, Paul Caron and I have kick off our series on unsolicited advice to Erwin Chemerinsky.  See TaxProf.   For readers interested in making a contribution, feel free to send me your 250 word post.   We are looking for students and practicing lawyers as well as academics. 

Indiana Lawyers and WSJ Article on Legal Job Market

The blogosphere is abuzz with Amir Efrati's excellent WSJ page-1 story on the changing legal job market (see TaxProf, Prawfsblawg, Legal Ethics Blog, Ideoblog, and Leiter).  Afrati's story took a lot of factual investigation, and I applaud him for tracking down so many relevant sources.  Prospective students will benefit.  In addition, the ABA is sure to take notice. 

One of the graphics in the WSJ story was based on my recent survey of ISBA lawyers, though you have to dig to find it.  Thus I am posting it here.  This chart is a breakdown of change in lawyer income over the last five years (excluding retiring lawyers), broken down by respondent's firm size:

New ICPSR Data Releases

Here.  These two caught my eye:

• 2778 Gambling Impact and Behavior Study, 1997-1999: [United States]

• 4697 General Social Surveys, 1972-2006 [Cumulative File]  [United States] [updated file]

IRB Approved?

I'm not sure if this research -- which is supported by the National Highway Traffic Safety Administration -- is subject to federal human subjects rules, but I wonder if any IRB would actually approve this method of recruiting participants:

The Gilpin County Sheriff's Office was apologizing Monday after a weekend effort to help a research group led to complaints about what appeared to be a DUI checkpoint - but wasn't.

Sheriff's officials who participated in the stops now acknowledge that the nonprofit organization requesting voluntary DUI and drug tests from drivers was overly persistent, according to complaints.

"It was like a telemarketer that you couldn't hang up on," said Gilpin County Undersheriff John Bayne.       

Sgt. Bob Enney said deputies assisted the Pacific Institute for Research and Evaluation in stopping motorists at five sites along Colorado 119 for surveys on any drug and alcohol use. Surveyors then asked the motorists to voluntarily submit to tests of their breath, blood and saliva. At least 200 drivers were tested, Enney said. About five motorists later complained, he said.

Roberto Sequeira, 51, said he and his wife, Terry, were detained for 15 minutes Friday evening despite their protestations that they needed to get their sleepy 10-year-old child back to their home in Nederland.

. . . .

Christopher N. Osher, Alcohol Surveys Spur Complaints, Denver Post, Sept. 18, 2007.

Advice for Erwin Chemerinsky

This was a good week for me because Erwin Chemerinsky accepted the deanship at UC Irvine.  I have never met Erwin and, beyond his excellent reputation as a scholar, have no special knowledge of his skills and abilities.   But UC Irvine had (and once again has) it all: brand, in-state tuition, location, resources, and a whistle clean slate.  When the UC system hired and fired him all in the course of week, I was convinced that the last, best chance to do something truly unique in legal education--with a reasonable shot a success and wider influence--had been destroyed. 

Amidst the enormous national attention generated by the Chemerinsky episode (see roundup of coverage, see here.), Paul Caron and I want to use this moment to generate and publicize the best ideas about reforming legal education from some of the leading thinkers in the law school world. Next week, TaxProf Blog will begin posting the answers of university presidents and provosts, and law school deans, faculty, students, and career planning professionals, to this question:

What is the single best idea for reforming legal education you would offer to Erwin Chemerinsky as he builds the law school at UC-Irvine?

Each contribution will be limited to no more than 250 words (no exceptions).  Although we are soliciting contributions, I would be happy to consider unsolicited postings, particularly from students and practicing lawyers.  You can email me here (no guarantees, however, that Paul and I will use it).

Appointed and Elected Judges: An Empirical Comparison

A recent paper by Stephen Choi (NYU), Mitu Gulati (Duke), and Eric Posner (Chicago) will surely interest judicial decisionmaking scholars (and others). In Professionals or Politicians: The Uncertain Empirical Case for an Elected Rather than Appointed Judiciary, the authors set out to test the conventional wisdom that suggests "appointed judges are superior to elected judges because appointed judges are less vulnerable to political pressure." Drawing on a dataset of state high court opinions, the authors "construct objective measures for three aspects of judicial performance: effort, skill and independence. The measures permit a test of the relationship between performance and the four primary methods of state high court judge selection: partisan election, non-partisan election, merit plan, and appointment." A brief description of the results from their study follows.

"... results do not show appointed judges performing at a higher level than their elected counterparts. Appointed judges write higher quality opinions than elected judges do, but elected judges write many more opinions, and the evidence suggests that the large quantity difference makes up for the small quality difference. In addition, elected judges do not appear less independent than appointed judges. The results suggest that elected judges are more focused on providing service to the voters (that is, they behave like politicians), whereas appointed judges are more focused on their long-term legacy as creators of precedent (that is, they behave like professionals)."

Michael Dorf on Moneyball

In my last post, I had a throw away line on Columbia law professor Michael Dorf.  For those following the link I provided, the clear implication was that Dorf's comments on the Chereminsky affair -- e.g., "[e]ven solid but middling-ranked law schools can have at best a marginal impact on the course of legal education" -- was evidence of Dorf's overconfidence in his own judgment.

Dorf picked up on the Moneyball theme in the comments section, claiming that he in no way endorsed the data-blind approaches of traditional baseball scouts.   Professor Dorf then extended the baseball analogy to buttress his underlying point: 

“[A] discerning reader will see [from Dorf's original post and subsequent qualifying comments] that I was making a quite different point:  I was guessing [to some of us, it looked like concluding] that elite law schools could do a middling job at actually educating students and still not suffer serious reputational hits, because their students' talents cover up for this. (This is why the Yankees have been a good team even though they didn't follow the path of Billy Bean: their huge war chest covered up their errors.) I don't deny that eventually good innovations win out but I say that it will likely require a market leader to make the process work. Thus, the Boston Red Sox have been more successful using Bean's strategy than the A's were, because the Sox have done it with more money.

Sure, money / starting position matters.  But the fact remains that virtually all major league clubs now employ the techniques originally created by a small market ("middling-ranked") team.  Far from having a "marginal impact",  these innovations revolutionized baseball.

Starting with a clean slate, could Chemerinsky do the same at UC Irvine?  I suspect a large number of deans would love to play his hand:  (1) the UC brand and faculty benefits, (2) a public tuition subsidy for students, (3) a spectacular geographic location adjacent to a vibrant legal market, (4) the ability to select an entire faculty based on a unique institutional vision, and (5) seed money from private donors. 

In our empirical work on law school rankings (here and here), Andy Morriss and I found that (a)  proximity to large legal markets and (b) low cost (e.g., public school or low avg debt) are both drivers of higher LSAT scores over time.  UC Irvine has both of these advantages.   Further, if UC Irvine can document that its innovative teaching adds more value in three years (a LONG time) than its tradition-bound competitors, then employers (and eventually students) will move the market.  How elite is a law school when students and employers find a better value downstream?

At present, we do not know the potential benefits of innovative teaching methods on human capital--e.g., can the gains be permanent and persistent, trumping raw ability (as measured by LSAT scores, far from perfect) over time?  This is an important empirical question.  I know that some legal educators assume the answer is no.  Because the stakes are so high, I would prefer to see the data.   

If law schools -- during the course of three years -- can add substantial and persistent value beyond raw aptitude, the innovators (and I hope Chemerinsky is one of them) could get a huge competitive advantage, at least until the elite schools are pressured by market forces to learn and apply those same innovations.  And this would be far from a "marginal impact".

NY Times Review of Super Crunchers

This Sunday's N.Y. Times had an interesting review of Ian Ayres new book, Super Crunchers: Why Thinking by Numbers is the New Way to Be Smart.   In a nutshell, this book piles on to a theme developed by Moneyball and Freakanomics -- you can profit by data mining, especially when the conventional wisdom is wrong.   

NY Times economics reporter, David Leonhardt, begins his review of Super Crunchers with an anecdote on Orley Ashenfelter's theory that the weather in Bordeaux is the best predictor of a wine's eventual price.  Apparently, Ashenfelter's theory irritated the wine establishment, which heaped scorn upon him.  In fact, Ashfelter is part of lore of the early days of the American Law & Economics Associations.  Many years ago, Ashenfelter gave the ALEA key note address over dinner and entertained the crowd with his musings on weather and wine prices, noting (to the delight of his listeners) that his theories had already produced a tidy profit.

To my mind, the principles of Moneyball-Freakanomics-Super Crunchers are hard to dismiss. Indeed, over the last several years, I cannot recall a single frontal attack on any of these ideas.  Instead, it is my observation that most people (and thus organizations and institutions) never take the time to think through their simple, stark implications--i.e., don't be complacent, look at the data, or eventually you are going to get smoked by the competition.   

Here is the key passage in Leonhardt's review:

"For all its successes ... statistical analysis continues to face tremendous skepticism and even animosity. For one thing, Ayres notes, statistics threaten the 'informational monopoly' of experts in various fields. But even to many people without a vested interest, relying on cold, hard numbers rather than human instinct seems soulless.

The problem, as Aryes points out, is that there is overwhelming evidence that human beings are notoriously overconfident in their own judgments and intuitions.  Indeed, ever since Moneyball, every where I turn I am astonished by the overconfidence of the well-placed and the well-credentialed (today's winner being Michael C. Dorf).  This creates a lot of opportunities for the rest of us.   

When Do Justices Use Legislative History

James Brudney of the Ohio State University Moritz College of Law and Corey Ditslear of the University  of North Texas political science department, have posted a new piece on SSRN.  Entitled Liberal Justices' Reliance on Legislative History: Principle, Strategy, and the Scalia Effect, it addresses the question of whether legislative history is used ideologically, in particular by the liberal justices.  The abstract follows:

This article conducts an in-depth examination of Supreme Court Justices' reliance on legislative history during the Burger, Rehnquist, and early Roberts eras. In doing so, it makes two important contributions to current statutory interpretation debates.
First, the article presents a powerful case against the conventional wisdom that legislative history is a "politicized" resource, invoked opportunistically by federal judges. The premise that judges regularly rely on legislative history to promote their preferred policy positions - if true - should find ample support in the majority opinions of liberal Supreme Court Justices construing liberal (pro-employee) labor and civil rights statutes. By analyzing all 320-plus majority opinions in workplace law authored by eight liberal Justices from 1969-2006, the authors establish that legislative history reliance is actually associated with a constraining set of results. When these eight liberals use legislative history as part of their majority reasoning, they do so to justify a higher proportion of their pro-employer outcomes than their pro-employee decisions. The authors then review individual majority opinions to demonstrate how this surprising pattern of reliance is based on neutral doctrinal considerations. Liberal Justices use legislative history to illuminate the existence and contours of complex statutory bargains that often favor conservative or pro-employer positions. The authors consider alternative explanations, premised on the institutional factor of who assigns majority opinions and also the instrumental possibility that liberals withhold use of legislative history in "minor" cases to enhance its value in more important decisions. They conclude, however, that Justices Brennan, Marshall, Souter, Stevens, and others are willing to follow so frequently a legislative history trail leading away from their presumed ideological preferences mainly because they have invoked this interpretive resource in principled fashion.
The article's second major contribution is to identify and analyze the Scalia Effect that has arisen with respect to liberal Justices' use of legislative history since 1986. In the face of Justice Scalia's fervently expressed opposition to legislative history, liberal Justices have opted not to rely on that resource in a series of pro-employer majorities that Scalia joins. One result of the liberals' strategic restraint is to make their use of legislative history in remaining (mostly pro-employee) majority opinions appear more ideological than was true before Scalia joined the Court. The authors also show that liberal justices have special reasons for acting strategically in this regard. When liberals rely on legislative history, Justice Scalia is significantly less likely to join their majority opinions even when he votes on their side; he also is significantly less likely to vote for the majority result when these liberals rely on legislative history than when they do not. Intriguingly, Justice Scalia's strong resistance to legislative history usage does not extend to majorities authored by his conservative colleagues. Scalia seems prepared to give these conservative colleagues a free ride: he is every bit as likely to join their majorities, or vote for their results, when they rely on legislative history as when they do not.

(Hat tip to Rick Hasen's legislation listserv.)

The authors' discussion of the "Scalia effect" reminds me of a moment in Cass Sunstein's Elements class in my first year of law school.  Beware of what conclusions you draw from observations, he said.  Back in the 1960s, for example, many country clubs issued new rules requiring men to wear coats and ties.  But this was not because there was a move towards more formality.  Rather it was the opposite.  Social norms were moving away from formality and the clubs that wanted to preserve the old norms found it necessary, for the first time, to explicitly state them.  Not quite analogous to the Scalia effect discussion, but both are an important reminder of the hazards of inferring causation.

Judicial Decisions & Ideology North of the (American) Border

Prompted partly by Cass Sunstein's study of the influence of a judge's ideology on court of appeals decisions, Moin Yahya (Alberta, Law) and James Stribopoulos (Osgoode Hall, Law) undertake a similar study of "the busiest appellate court in Canada, the Court of Appeal for Ontario." In Does a Judge's Party of Appointment or Gender Matter to Case Outcomes? An Empirical Study of the Court of Appeal for Ontario (Canada), the authors find that:

"... at least in certain categories of cases, both party of appointment and gender are statistically significant in explaining case outcomes. Between these two variables, gender actually appears to be the stronger determinant of outcome in certain types of cases."

Venturing beyond the empirics, the authors also argue that their "study also points toward a simple solution. Diversity in the composition of appeal panels both from the standpoint of gender and party of appointment dampened the statistical influence of either variable. In other words, in the case of gender, a single judge on a panel who is of the opposite sex from the others, or in the case of political party, a single judge appointed by a different political party, is sufficient to eliminate the potential distorting influence of either variable."

Remembering Fundamentals

A recent post over at the Statistical Modeling, Causal Inference, and Social Science blog provides a helpful reminder about one steadfast basic for empirical researchers--data plotting. I am constantly surprised (stunned, actually) by the number of researchers that neglect the fundamentals. It is difficult to overstate the importance of attending to the simple stuff.

Postscript on the Bi-Modal Distribution

We got a lot of traffic last week due to our post on the bi-modal salary distribution.   Many readers observed that Class of 2006 non-respondents are probably disproportionately lower earners because (a) they are reluctant to divulge embarrassing information, perhaps because they blame themselves, and (b) law schools are not particularly anxious to get salary information that will pull down their averages.   So, as bleak as the picture looks, it may be even worst.  I think this is a reasonable interpretative point.

Richard Matasar, the Dean at New York Law School (and author of the terrific essay "The Rise and Fall of the American Legal Education"), has an even better vantage point to evaluate these issues.  In an email exchange, Richard made the following observations [which he permitted me to post]:

The problem you highlight is significantly understated by two additional things.  First, the debt loads for the students in the two parts of the distribution are nearly identical and depending on how "merit" money is distributed, the debt may actually be higher for those in the bottom of the salary pool. [This is because at each school, higher credentialed students are more likely to (a) get merit aid, and (b) get the grades necessary to get a large firm job.  WDH]. 

Second, NALP also has longer term data that reveal that the gap widens over time.  This is confirmed in part by the [After the JD Study] that talks about initial difference, ties it to schools, and then looks out over time.  And, as you know, the distribution looks similar at schools throughout the hierarchy outside of a very small group at the top [i.e., national law schools, which will likely have a single mode that is very high, such as $160K].

This is serious stuff.  In Rise and Fall, Matasar observed that "we may be reaching the end of a golden era for law schools, beginning a period of decline, and putting many schools' survival at risk."

The Empirics of Taking the Stand in Criminal Cases

Two of my esteemed Cornell Law School colleagues (and, in the interest of full disclosure, very good friends as well), Ted Eisenberg and Valerie Hans, recently circulated a paper that will likely interest, among others, criminal proceduralists.  In Taking a Stand on Taking the Stand: The Effect of a Prior Criminal Record on the Decision to Testify and On Trial Outcomes, Ted and Valerie exploit a wonderful dataset and assess "relations between the existence of a prior criminal record and defendants testifying at trial, between testifying at trial and juries' learning about a criminal record, and between juries' learning about a criminal record and their decisions to convict or acquit." A summary of their findings, drawn from their abstract, follows.

"For cases with strong evidence against defendants, learning of criminal records is not strongly associated with conviction rates. Juries appear to rely on criminal records to convict when other evidence in the case normally would not support conviction. Use of prior record evidence may therefore lead to erroneous convictions."

APSA Award

I would like to congratulate our own Sara Benesh, who received the 2007 Houghton-Mifflin award from the "Law and Courts" section of the American Political Science Association for her paper entitled "Understanding Public Confidence in American Courts."  According to the guidelines, The Houghton-Mifflin Award recognizes the best journal article on law and courts written by a political scientist and published during the previous calendar year.  The piece was published in the August 2006 issue of the Journal of Politics.  Congratulations Sara.

Hat Tip: Jeff Yates of Voir Dire (see post here)

Distribution of 2006 Starting Salaries: Best Graphic Chart of the Year

The most recent edition of NALP's serial publication,  Jobs & JD's, includes the chart excerpted below [click-on to enlarge].  It is the distribution of full-time salaries for all members of the Class of 2006 who reported income data to their respective law school (22,665 graduates).  If you were looking for a single graphic to illustrate the most vexing problems facing law firms, law students, and law schools, this would be it.  A more dramatic bimodal distribution you will not find.

The sample includes--in order of size--private practice (55.8%), business (14.2%), government (10.6%), judicial clerks (9.6%), public interest (5.4%), and other (2.8%).  Half of the graduates make less than the $62,000 per year median--but remarkably, there is no clustering there.  Over a quarter (27.5%) make between $40k-$55k per year, and another quarter (27.8%) have an annual salary of $100K plus.   

If the chart were a flipbook of the last twenty years, the first mode would be relatively stationary, barely tracking inflation, while the second mode would be moving quickly to the right--i.e., the salary wars.  In fact, because of the recent jump to $160K in the major markets, the second mode has already moved even more to the right.

What are the implications of this chart?

• For law students. Let's face it:  $40K to $55K per year is just not enough to pay down the avg. $85,000 debt (especially as interest rates climb) and still enjoy any kind of lifestyle that a professional degree is presumed to confer. The national median starting salary for a 2 to 10 lawyer firm is $50,000.   There are a lot of struggling alumni out there. And do we really need more law schools?  For many, getting a JD is a very risky financial proposition, especially when you factor in bar passage.

• For law schools.  Because different law schools supply graduates into different modes (roughly tracking US News rank), it is indisputable that lower-ranked schools cannot continue to heap ever higher debt onto their students.  On the other hand, if you are Georgetown, NYU, Northwestern, Harvard, Columbia, et al., your current model works just fine.  The salary wars make this possible.    Yet, 50 to 75 other law schools raising tuition in order to buy their way into the Top 15 is a classic positional competition--and it is socially harmful, with our students bearing the cost.  As legal educators, we can do better.

• For law firms. The second mode keep moving to the right because too many law firms refuse to reconsider their business model in light of a continuing surge in demand for corporate legal services.   All these firms want Harvard, Columbia, Chicago graduates, etc. and, if necessary, Illinois (top 25%),  Indiana (top 15%), Marquette (top 10%), etc.  If legal education worked like any other market, Northwestern would be merging with Cardozo, exploiting Cardozo's capacity and location and leveraging Northwestern's brand.  But law schools are maximizing prestige, not output or profit.  Such a merger would be dilutive or break-even at best for the higher ranked school.

There is a lot of commodity corporate legal work on there; why not bow out of the salary wars, ratchet down the hours to 1800, take work on a flat fee arrangement, focus on better/faster service (thus increasing margins on the flat fees), and literally feast on the human capital willing to take a job in the "death valley" range (i.e., ~$80,000 per year), especially if the hours are sane.  The client gets quality and cost predictability, and the well-managed firm can make a lot of money.  This is a great opportunity for a firm willing to rethink its business model. Larry Ribstein's publicly held law firm would be all over this. But any established large firm willing to think outside the box could do it.

Challenges To (or Perils of) Empirical Research

Over at MoneyLaw Jeff Harrison (Florida) shares some of the challenges incident to his desire to study, empirically, "the scholarly productivity of graduates of elite schools with that of graduates of non elite schools." If nothing else, his musings perhaps remind us anew that, in the world of empirical legal scholarship, precious little comes easily.