Empirical Legal Studies

Federal Judges' Citations to Legal Scholarship

Derogating legal scholarship has become something of a sport for many, including federal judges. Chief Justice Roberts, for example, recently opined that "because law review articles are not of interest to the bench," he has trouble remembering the last law review article he read.

David Schwartz (Chicago-Kent) and Lee Petherbridge (Loyola-LA) subject the general claim to data. In a series of papers the authors present findings on when an opinion (majority, dissent, or concurrence) cites to legal scholarship in the U.S. Supreme Court, Courts of Appeals, and Federal Circuit. For Supreme Court opinions, the authors find that legal scholarship citations "sharply vary across different types of legal issues." Click here for a quick summary of the papers (and the data sets).

Empirical Legal Scholars and "Special" Responsibilities

A recent post by David Schwartz (Chicago-Kent)--wondering whether empirical legal scholars should shoulder "special ethical responsibilities"--ignited a fascinating (and timely) discussion over at Concurring Opinions. Two reasons prompt Schwartz's concerns. "First, nearly all law reviews lack formal peer review. The lack of peer review potentially permits dubious data to be reported without differentiation alongside quality data. Second, empirical legal scholarship has the potential to be extremely influential on policy debates because it provides 'data' to substantiate or refute claims. Unfortunately, many consumers of empirical legal scholarship — including other legal scholars, practitioners, judges, the media, and policy makers — are not sophisticated in empirical methods."

Schwartz's concern focuses on what he calls "weak data." By that he means "reporting [results from] data that encourages weak or flawed inferences, that is not statistically significant, or that is of extremely limited value and thus may be misused." Specifically, "[t]he precise question I have been considering is under what circumstances one should report weak data, even with an appropriate explanation of the methodology used and its potential limitations."

Whether you agree with Schwartz or not, he raises an important question that warrants attention.

Helpful Reminders on How To Minimize Errors in Empirical Research

Responding to the recent debacle involving a grad student uncovering a blundering error in a paper by noted Harvard economists (here), Betsey Stevenson (Mich.) and Justin Wolfers (Mich.) initiated a (now growing) list of suggestions on how to minimize errors in empirical research. Not surprisingly, others, including Andrew Gelman (Columbia), have added to the list (here). While the list will inevitably grow, it already includes basic, helpful reminders for even the most experienced researchers.

Replication, Replication, Replication

A recent news story underscores the importance of basic replication (as well as scholarly attention to detail) for empiricists.

"His [Thomas Herndon's] professors at the University of Massachusetts-Amherst had set his graduate class an assignment--pick an economics paper and see if you can replicate the results. It's a good exercise for aspiring researchers. Thomas chose Growth in a Time of Debt. It was getting a lot of attention, but intuitively, he says, he was dubious about its findings."

Turns out that the grad student's intuition was dead-on as core results from the influential economics article--authored by two leading Harvard economists--could not be replicated. Herndon's replication efforts uncovered a basic error in the spreadsheet. "The Harvard professors had accidentally only included 15 of the 20 countries under analysis in their key calculation (of average GDP growth in countries with high public debt). Australia, Austria, Belgium, Canada and Denmark were missing." In addition, other data for some countries were missing altogether.

Who "Owns" History?

While questions about who owns judges' official working papers implicate legal historians most directly, such (admittedly complex) questions should also interest empirical legal historians. In "Judges and Their Papers," Kathryn Watts (Washington) makes the case that judicial papers should be construed as public rather than private property. An excerpted abstract follows.

"This Article is the first to give significant attention to the question of who should own federal judges’ working papers and what should happen to the papers once a judge leaves the bench. Upon the 35th anniversary of the enactment of the Presidential Records Act, this Article argues that judges’ working papers should be treated as governmental property — just as presidential papers are. Although there are important differences between the roles of President and judge, none of the differences suggest that judicial papers should be treated as a species of private property. Rather, the unique position of federal judges, including the judiciary’s independence, should be taken into account when crafting rules that speak to reasonable access to and disposition of judicial papers — not when answering the threshold question of ownership. Ultimately, this Article — giving renewed attention to a long forgotten 1977 governmental study commissioned by Congress — argues that Congress should declare judicial papers public property and should empower the judiciary to promulgate rules implementing the shift to public ownership. These would include, for example, rules governing the timing of public release of judicial papers. By involving the judiciary in implementing the shift to public ownership, Congress would enhance the likelihood of judicial cooperation, mitigate separation of powers concerns, and enable the judiciary to safeguard judicial independence, collegiality and confidentiality."

USN 2014 Rankings

As you've probably heard, the U.S. News 2014 Law School Rankings are out. Rather than offer commentary, I thought I'd piggyback on Paul Caron's useful post comparing the overall rankings with the peer reputation rankings. So here, for your edification, are the numbers Paul compiled in scatterplot form. (PDF)

The Empirics of Regulation

Now that Cass Sunstein (Harvard) has departed the Obama Administration (and OIRA) and migrated back to academic life, in a recent paper published by the University of Chicago Law Review, Empirically Informed Regulation, Sunstein illustrates the central role data play (or, at least should play) in the development of regulations, with an emphasis on behavioral economics. The paper's abstract follows.

"In recent years, social scientists have been incorporating empirical findings about human behavior into economic models. These findings offer important insights for thinking about regulation and its likely consequences. They also offer some suggestions about the appropriate design of effective, low-cost, choice-preserving approaches to regulatory problems, including disclosure requirements, default rules, and simplification. A general lesson is that small, inexpensive policy initiatives can have large and highly beneficial effects. In the United States, a large number of recent practices and reforms reflect an appreciation of this lesson. They also reflect an understanding of the need to ensure that regulations have strong empirical foundations, both through careful analysis of costs and benefits in advance and through retrospective review of what works and what does not."

Unions and Worker Safety

Allison Morantz (Stanford) has an interesting paper, Coal Mine Safety: Do Unions Make a Difference?, in the current Industrial and Labor Relations Review, Vol. 66:1 (Jan. 1, 2013). The paper draws from UMWA data and finds that increased unionization correlates with a decrease in serious worker injuries, but an increase in nontraumatic worker injuries. Perhaps even more interesting, however, is the potential effect of unionization on worker injury reporting. The abstract follows.

"Although the United Mine Workers of America (UMWA) has always advocated strongly for miners' safety, prior empirical literature contains no evidence that unionization reduced mine injuries or fatalities during the 1970s and 80s. This study uses a more comprehensive dataset and updated methodology to examine the relationship between unionization and underground, bituminous coal mine safety from 1993 to 2010. I find that unionization predicts a substantial and significant decline in traumatic injuries and fatalities, the two measures that I argue are the least prone to reporting bias. These disparities are especially pronounced among larger mines. My best estimates imply that overall, unionization is associated with a 13-30% drop in traumatic injuries and a 28-83% drop in fatalities. Yet unionization also predicts higher total and nontraumatic injuries, suggesting that injury reporting practices differ between union and nonunion mines."

"Boys to the Back of the Class"

A provocative new study, Non-cognitive Skills and the gender Disparities in test Scores and Teacher Assessments: Evidence from Primary School, brings evidence to bear on long-assumed convention wisdom involving boys, girls, and academic performance. And the paper's findings raise quite difficult questions, including those for Title IX.

Univ. of Georgia economists Christopher Cornwell, David Mustard, and Jessica Van Parys set out to, in part, "examine the relationship between the (objective) test-score differences and (subjective) teacher grades." What they find is that while girls receive systematically higher grades than boys, on average, in elementary school, "the grades awarded by teachers are not aligned with test scores." The paper's abstract puts the point more bluntly: "Boys who perform equally as well as girls on reading, math and science tests are graded less favorably by their teachers." The authors ascribe the "misalignment" of  objective test scores and subjective teacher grades to "non-cognitive skill development" differences between elementary school boys and girls.

Not surprisingly, this academic article, published in the Journal of Human Resources (48:1, Winter 2013), quickly attracted an array of robust public commentary (e.g., here and here). Left un-plumbed, thus far anyway, are the potential implications for Title IX.

To be sure, causal directions in this study are anything but clear thus far. After all, when seeking to explain statistical "misalignment" between standardized test scores and teacher (non-anonymous) grades, whether boys over-perform on standardized tests or under-perform in classroom grades (or the reciprocal for girls) is anything but clear. Nonetheless, these findings raise potentially uncomfortable questions.

A Decade of JELS -- 10:1 (Mar. 2013)

As a JELS co-editor I note with both pride and sheer amazement the publication of Vol. 10. The current issue (10:1, Mar. 2013) includes a provocative array of wonderful papers, including Barton & Moran's, Measuring Diversity on the Supreme Court with Biodiversity Statistics, and Mak, Sidman & Sommer's, Is Certiorari Contingent on Litigant Behavior? Petitioners' Role in Strategic Auditing. All of the papers are worth reading.