Now, evidently, biomedical research is beginning to confront replication problems of their own (click here and here for discussions). One factor that likely distinguishes biomedical research involves the magnitude of financial incentives and their influence.
Researchers using judge-level data will benefit from a new on-line judicial database created by the Free Law Project. Funded in part by NSF and foundation grants, and in conjunction with Elliot Ash (Princeton) and Bentley MacLeod (Columbia), the database currently includes biographical information on almost 8,500 federal and state judges. For more information, see the database's announcement (here) and the judge search engine (here).
I previously linked to discussion of a paper assessing the degree to which Bayes Theorem informs judicial decisions. To the extent that judges aspire to play an "umpire" role (and in a slight bow to opening day), I note Andrew Gelman's (Columbia--Statistics) recent post (here) discussing the argument (developed in this paper) that Bayes Theorem informs baseball umpires' decisions on whether to call a "ball" or "strike."
The Eleventh Annual Conference on Empirical Legal Studies (CELS), hosted by Duke Law School, is scheduled for November 18-19, 2016, in Durham, NC. CELS brings together scholars from an array of fields, including law, economics, political science, psychology, and policy analysis. Papers are selected through a peer-review process and each is assigned an individual discussant. CELS invites empirical papers that span all areas of empirical legal studies. Authors are encouraged to submit works-in-progress; however, submissions should be completed paper drafts that include main results. Submitted papers must also be unpublished (and are expected to be unpublished at the time of the conference). Paper submission deadline is July 10, 2016.
The empirical literature seeking to understand judicial outcomes continues to develop and mature. Judge Posner's book, How Judges Think (2010), places Bayes' Theorem front-and-center. In a qualitative study of judges (N=30), Jack Knight (Duke), Mitu Gulati (Duke), and David Levy (Duke, and a former federal judge) set out to assess what Posner's book assumes. In How Bayesian Are Judges?, the authors conclude: "not at all." The abstract follows.
"Richard Posner famously modeled judges as Bayesians in his book, "How Judges Think?". A key element of being Bayesian is that one constantly updates with new information. This model of the judge who is constantly learning and updating, particularly about local conditions, also is one of the reasons why the factual determinations of trial judges are given deference on appeal. But do judges in fact act like Bayesian updaters? Judicial evaluations of search warrant requests for probable cause provides an ideal setting to examine this question because the judges in this context have access to information on how well they did on their probabilistic calculations (the officers who conduct the search have to file, in every case, a "return" detailing what was found in their search). Based on detailed interviews with thirty judges our answer to the "How Bayesian are Judges?" question is: Not at all. The puzzle we are left with, given that acting in a Bayesian fashion is normal behavior for the rest of us, is why we get these puzzling results for judges in the search warrant context?"
In some instances a raw data set needs to be reshaped (from a rows-to-columns format or visa-versa) to facilitate focus on a desired unit of analysis. A helpful discussion on the Stata Forum (here) provides a brief explanation and useful code suggestions.
Schlegel's (1995) important and comprehensive treatment of American legal realism and "empirical social science" makes clear the importance of, among other factors, the "Brandeis Brief" in Muller v. Oregon (1908), Yale's comparatively under-appreciated Underhill Moore, and the Legal Realists, broadly understood, to today's empirical legal studies. Marty Wells (Cornell--Statistics) brought an additional fascinating piece of empirical legal studies history to my attention that warrants circulation as it also contributed to ELS's development. Specifically, in late 1912 Cornell professor (and American Statistical Association President) Walter Willcox's presidential address (subsequently published in 1913), entitled The Need of Social Statistics as an Aid to the Courts, notes: "It is the lack of convincing social statistics upon such problems which has made it impossible to answer with confidence many of the questions judges and legislatures have assumed or felt bound to answer."
"Of 25 firearm laws, nine were associated with reduced firearm mortality, nine were associated with increased firearm mortality, and seven had an inconclusive association. After adjustment for relevant covariates, the three state laws most strongly associated with reduced overall firearm mortality were universal background checks for firearm purchase (multivariable IRR 0·39 [95% CI 0·23–0·67]; p=0·001), ammunition background checks (0·18 [0·09–0·36]; p<0·0001), and identification requirement for firearms (0·16 [0·09–0·29]; p<0·0001)."
What the paper's skeptics (and critics) note include the results' strength and a healthy skepticism about laws' ability to independently account for change of such magnitude.
UPDATE: The paper's authors have quite helpfully responded publicly (here) to some of the critiques. One concern involves the stability of results flowing from a model that includes an N of 50 yet 25 predictors. As Kalesan et al. note in their response: "The critiques [partly] rest on the limitations of the sample size and on the model robustness. However, as we indicated in the paper, and in the accompanying statistical appendices, we ran an exhaustive set of sensitivity analyses and found the results robust to multiple challenges. In particular, we provide sensitivity analysis by using the change in gun death rates from 2008 to 2010 as the outcome and found results similar to the main results."
"Many jurisdictions award pain and suffering damages, yet it is difficult for judges or juries to quantify pain. Several jurisdictions, such as California, cap pain and suffering damages or other noneconomic damages, and legal scholars have proposed ways to control such damages. Reforms and proposals, however, have been based on limited empirical evidence. It remains an open question whether components of economic damages explain pain and suffering damages. This study employs a unique data set of Taiwan district court cases and uses detailed information on the components of pecuniary damages. Pain and suffering damages highly correlate with the plaintiff’s medical expenses, level of injury, and the amount requested by the plaintiff. The association with the amount requested by the plaintiff persists when one accounts for the likely quantifiable influences on pain and suffering damages, evidence of a possible anchoring effect. The strong correlation between economic damages and noneconomic damages persists in a large U.S. dataset of judge and jury trials, in which the noneconomic fraction of total damages is no greater than the pain and suffering fraction of total damages in Taiwan. Judges and juries consistently produce coherent patterns of noneconomic damages."
For those interested, I highly recommend the array of summer quant programs at University of Michigan's ICPSR. While geared toward political science, the spill-over for legal scholars is ample.
2016 ICPSR Summer Program in Quantitative Methods of Social Research Scholarships and Training for Researchers in Political Science
The ICPSR Summer Program offers rigorous, hands-on training in statistics, quantitative methods, and data analysis for students, faculty, researchers, and policy-makers across the social, behavioral, and medical sciences.
In 2016, the ICPSR Summer Program is offering more than 80 courses in cities across the U.S., Canada, and Europe on a range of topics, including Bayesian analysis, data reproducibility and transparency, game theory, text analytics, spatial econometrics, field experiments, and network analysis.
The ICPSR Summer Program also offers scholarships to support advanced graduate students and pre-tenure scholars in Political Science who wish to take courses in the Summer Program. The application deadline for the following scholarships is April 30, 2016.
No doubt partly prompted by a recent spate of academic fraud, Science magazine published results from a study that set out to replicate 100 published psychology studies. An "international team of experts," however, could "reproduce only 36% of original findings." (News coverage here.) While I'm sure many in the psychology field will seek to explain such findings, an inability to replicate 74% of studies published in "top" psychology journals is, at the very least, jarring.
Update 2: In a recent editorial in Psychology Science (here), D. Stephen Lindsay responds to a replication effort, published in Science (described above), that signaled serious problems. As Gelman (Columbia--Statistics) notes, "Lindsay talks about replication problems and how researchers should do better. He warns about p-hacking, noise, and the difference between significance and non-significance not being itself statistically significant." While it's notable to see a leading psychology journal recognize a problem and undertake concrete editorial policy changes to address the problem, it is likely only a (necessary) first step. To be sure, the "replication issue" is certainly not confined to psychology and it's an issue that warrants continued and sustained scholarly attention.
Update 3: From the 3.4.2016 NYT (here). Reporting on a paper critiquing the 2015 Science paper challenging replicability of 100 published psychology studies. Additional commentary from Gelman (Columbia--Statistics) here,here, and here.
Update 4: Marty Wells (Cornell--Statistics) notes that, according to a recent report in Science, replication problems have expanded into experimental economics.
On the theory that "timing is everything," Justin Pidot's (Denver) recent paper, Tie Votes in the Supreme Court, warrants attention in light of Associate Justice Scalia's death and the prospect of an 8-member Court for some period of time. Pidot's paper descriptively analyzes the 164 instances between 1925 and 2015 in which the a tie vote emerged from the Court and advances a change to existing Court policy. An excerpted abstract follows.
"This Article constitutes the first detailed empirical analysis of whether the Supreme Court’s current approach to tie votes makes sense, presenting an original study of the 164 instances in which a tie vote occurred between 1925 and 2015. Those data reveal two important trends, both of which suggest the current approach is at best unnecessary. First, where a case ends in a tie, the issue involved is either presented to the Supreme Court again in relatively short order or turns out to be of little significance. Second, only 1 of the 164 cases would today fall within the Supreme Court’s limited mandatory jurisdiction. The remainder would arrive at the Court on a writ of certiorari, a docket that is entirely discretionary.
Instead of affirming by equal division, the Supreme Court should utilize an alternate and equally well-established procedure to dismiss cases as improvidently granted. At a time when Americans disapprove of the Supreme Court at unprecedented levels, dismissing rather than formally announcing a tie to terminate deadlocked cases better protects the public perception of the Court’s legitimacy. Moreover, dismissal would reduce the potential for justices to write opinions that take public positions on issues that have not yet been resolved. Finally, as a matter of cognitive psychology, justices may feel internal pressure to remain consistent with a position once staked out in a vote that resolves a case. Dismissing, rather than affirming by an equally divided court, could alleviate that psychological pressures, allowing the justices to be more fair-minded when approaching cases in the future."
In a recent paper, Dawn of the Discipline-Based Law Faculty, Lynn LoPucki (UCLA) documents the sharp increase in entry-level hires possessing PhDs at AALS-member law schools between 2011 and 2015. According to Stephen Bainbridge (UCLA), who "deplores" this trend, "LoPucki believes the trend in favor of PhD law professors has reached a point of no return, potentially cementing a future where law school faculty, are long on degrees and publications—but short on time outside university walls." Brian Leiter (Chicago) has also recently weighed in on this trend, and his take is a bit more measured and detached: "Although I'm quoted saying that the rise of JD/PhDs will continue, that's a descriptive not normative statement. I think different schools have different missions. And the relevance of the JD/PhD varies by field."
While the rise in empirical legal scholarship and the concurrent increase in law faculty possessing a JD/PhD combo are distinct, to ignore how these two trends interact is to blink at reality. As such, legal empiricists need to be mindful of these critiques.
UPDATE: Readers pointed me to another recent paper, The Ph.D. Rises in American Law Schools, 1960-2011: What Does It Mean for Legal Education?, by Justin McCray (Berkeley) et al. In it, the authors study faculty hiring at the 34th highest-ranked law schools between 1960-2011. After documenting a rise in PhD hiring, the authors note that "the shift toward Ph.D.s entails a complex set of benefits and costs for law schools, and that there is the potential for building connections between practical experience and academic research, rather than simply choosing between them."
Northwestern and Duke Universities will hold their 7th annual week-long workshop on Research Design for Causal Inference, at Northwestern University from August 1-5, 2016. Organized by Bernie Black (Northwestern) and Mat McCubbins (Duke), the workshop features an outstanding faculty and is an excellent and efficient way to become acquainted with contemporary approaches for making causal inferences from various kinds of observational and experimental data. Details and registration information can be found here. A brief overview follows:
"Research design for causal inference is at the heart of a 'credibility revolution' in empirical research. We will cover the design of true randomized experiments and contrast them to natural or quasi experiments and to pure observational studies, where part of the sample is treated in some way, the remainder is a control group, but the researcher controls neither the assignment of cases to treatment and control groups nor administration of the treatment. We will assess the causal inferences one can draw from a research design, threats to valid inference, and research designs that can mitigate those threats."
Even the self-important newspaper of "record," the New York Times, could benefit from a basic statistics refresher. As Andrew Gelman (Columbia--Statistics) observed, a Feb. 11, 2016, NYT headline read as follows: "Gravitational Waves Detected, Confirming Einstein's Theory". Of course, as Gelman rightly points out, a more technically accurate headline would have read: "Gravitational Waves Detected, Not Falsifying Einstein's Theory".