Empirical Legal Studies

Reshaping Data

In some instances a raw data set needs to be reshaped (from a rows-to-columns format or visa-versa) to facilitate focus on a desired unit of analysis. A helpful discussion on the Stata Forum (here) provides a brief explanation and useful code suggestions.

ELS History

Schlegel's (1995) important and comprehensive treatment of American legal realism and "empirical social science" makes clear the importance of, among other factors, the "Brandeis Brief" in Muller v. Oregon (1908), Yale's comparatively under-appreciated Underhill Moore, and the Legal Realists, broadly understood, to today's empirical legal studies. Marty Wells (Cornell--Statistics) brought an additional fascinating piece of empirical legal studies history to my attention that warrants circulation as it also contributed to ELS's development. Specifically, in late 1912 Cornell professor (and American Statistical Association President) Walter Willcox's presidential address (subsequently published in 1913), entitled The Need of Social Statistics as an Aid to the Courts, notes: "It is the lack of convincing social statistics upon such problems which has made it impossible to answer with confidence many of the questions judges and legislatures have assumed or felt bound to answer."

When Results Might Be "Too Good To Be True"

An interesting post (and discussion) over at Andrew Gelman's (Columbia--Statistics) blog drills down on a recent paper published in The Lancet. The paper, Firearm legislation and firearm mortality in the USA: a cross-sectional, state-level study, by Bindu Kalesan et al., finds that:

"Of 25 firearm laws, nine were associated with reduced firearm mortality, nine were associated with increased firearm mortality, and seven had an inconclusive association. After adjustment for relevant covariates, the three state laws most strongly associated with reduced overall firearm mortality were universal background checks for firearm purchase (multivariable IRR 0·39 [95% CI 0·23–0·67]; p=0·001), ammunition background checks (0·18 [0·09–0·36]; p<0·0001), and identification requirement for firearms (0·16 [0·09–0·29]; p<0·0001)."

What the paper's skeptics (and critics) note include the results' strength and a healthy skepticism about laws' ability to independently account for change of such magnitude.

UPDATE: The paper's authors have quite helpfully responded publicly (here) to some of the critiques. One concern involves the stability of results flowing from a model that includes an N of 50 yet 25 predictors. As Kalesan et al. note in their response: "The critiques [partly] rest on the limitations of the sample size and on the model robustness. However, as we indicated in the paper, and in the accompanying statistical appendices, we ran an exhaustive set of sensitivity analyses and found the results robust to multiple challenges. In particular, we provide sensitivity analysis by using the change in gun death rates from 2008 to 2010 as the outcome and found results similar to the main results."

Predicting Pain & Suffering Damages From Economic Harm: Taiwan Data

A recent paper by Yun-chien Chang (Academica Sinica [Taiwan]) et al., Pain and Suffering Damages in Personal Injury Cases: An Empirical Study, explores potential correlates of pain and suffering awards. What sets this interesting paper apart is that it exploits a unique data set of Taiwan district court cases. The abstract follows.

"Many jurisdictions award pain and suffering damages, yet it is difficult for judges or juries to quantify pain. Several jurisdictions, such as California, cap pain and suffering damages or other noneconomic damages, and legal scholars have proposed ways to control such damages. Reforms and proposals, however, have been based on limited empirical evidence. It remains an open question whether components of economic damages explain pain and suffering damages. This study employs a unique data set of Taiwan district court cases and uses detailed information on the components of pecuniary damages. Pain and suffering damages highly correlate with the plaintiff’s medical expenses, level of injury, and the amount requested by the plaintiff. The association with the amount requested by the plaintiff persists when one accounts for the likely quantifiable influences on pain and suffering damages, evidence of a possible anchoring effect. The strong correlation between economic damages and noneconomic damages persists in a large U.S. dataset of judge and jury trials, in which the noneconomic fraction of total damages is no greater than the pain and suffering fraction of total damages in Taiwan. Judges and juries consistently produce coherent patterns of noneconomic damages."

Summer at ICPSR (and Mich.)

For those interested, I highly recommend the array of summer quant programs at University of Michigan's ICPSR. While geared toward political science, the spill-over for legal scholars is ample.

***

2016 ICPSR Summer Program in Quantitative Methods of Social Research
Scholarships and Training for Researchers in Political Science

The ICPSR Summer Program offers rigorous, hands-on training in statistics,
quantitative methods, and data analysis for students, faculty, researchers,
and policy-makers across the social, behavioral, and medical sciences.

In 2016, the ICPSR Summer Program is offering more than 80 courses in cities
across the U.S., Canada, and Europe on a range of topics, including Bayesian
analysis, data reproducibility and transparency, game theory, text
analytics, spatial econometrics, field experiments, and network analysis.

For a full schedule and course descriptions, visit: icpsr.umich.edu/sumprog           or contact: [email protected]

The ICPSR Summer Program also offers scholarships to support advanced
graduate students and pre-tenure scholars in Political Science who wish to
take courses in the Summer Program. The application deadline for the
following scholarships is April 30, 2016.

Replication Problems in Psychology (& Beyond)

No doubt partly prompted by a recent spate of academic fraud, Science magazine published results from a study that set out to replicate 100 published psychology studies. An "international team of experts," however, could "reproduce only 36% of original findings." (News coverage here.) While I'm sure many in the psychology field will seek to explain such findings, an inability to replicate 74% of studies published in "top" psychology journals is, at the very least, jarring.

Update: From the 8.27.2015 NYT (here).

Update 2: In a recent editorial in Psychology Science (here), D. Stephen Lindsay responds to a replication effort, published in Science (described above), that signaled serious problems. As Gelman (Columbia--Statistics) notes, "Lindsay talks about replication problems and how researchers should do better. He warns about p-hacking, noise, and the difference between significance and non-significance not being itself statistically significant." While it's notable to see a leading psychology journal recognize a problem and undertake concrete editorial policy changes to address the problem, it is likely only a (necessary) first step. To be sure, the "replication issue" is certainly not confined to psychology and it's an issue that warrants continued and sustained scholarly attention.

Update 3: From the 3.4.2016 NYT (here). Reporting on a paper critiquing the 2015 Science paper challenging replicability of 100 published psychology studies. Additional commentary from Gelman (Columbia--Statistics) here, here, and here.

Update 4: Marty Wells (Cornell--Statistics) notes that, according to a recent report in Science, replication problems have expanded into experimental economics.

SCOTUS Tie Votes

On the theory that "timing is everything," Justin Pidot's (Denver) recent paper, Tie Votes in the Supreme Court, warrants attention in light of Associate Justice Scalia's death and the prospect of an 8-member Court for some period of time. Pidot's paper descriptively analyzes the 164 instances between 1925 and 2015 in which the a tie vote emerged from the Court and advances a change to existing Court policy. An excerpted abstract follows.

"This Article constitutes the first detailed empirical analysis of whether the Supreme Court’s current approach to tie votes makes sense, presenting an original study of the 164 instances in which a tie vote occurred between 1925 and 2015. Those data reveal two important trends, both of which suggest the current approach is at best unnecessary. First, where a case ends in a tie, the issue involved is either presented to the Supreme Court again in relatively short order or turns out to be of little significance. Second, only 1 of the 164 cases would today fall within the Supreme Court’s limited mandatory jurisdiction. The remainder would arrive at the Court on a writ of certiorari, a docket that is entirely discretionary.

Instead of affirming by equal division, the Supreme Court should utilize an alternate and equally well-established procedure to dismiss cases as improvidently granted. At a time when Americans disapprove of the Supreme Court at unprecedented levels, dismissing rather than formally announcing a tie to terminate deadlocked cases better protects the public perception of the Court’s legitimacy. Moreover, dismissal would reduce the potential for justices to write opinions that take public positions on issues that have not yet been resolved. Finally, as a matter of cognitive psychology, justices may feel internal pressure to remain consistent with a position once staked out in a vote that resolves a case. Dismissing, rather than affirming by an equally divided court, could alleviate that psychological pressures, allowing the justices to be more fair-minded when approaching cases in the future."