Now, evidently, biomedical research is beginning to confront replication problems of their own (click here and here for discussions). One factor that likely distinguishes biomedical research involves the magnitude of financial incentives and their influence.
I previously linked to discussion of a paper assessing the degree to which Bayes Theorem informs judicial decisions. To the extent that judges aspire to play an "umpire" role (and in a slight bow to opening day), I note Andrew Gelman's (Columbia--Statistics) recent post (here) discussing the argument (developed in this paper) that Bayes Theorem informs baseball umpires' decisions on whether to call a "ball" or "strike."
No doubt partly prompted by a recent spate of academic fraud, Science magazine published results from a study that set out to replicate 100 published psychology studies. An "international team of experts," however, could "reproduce only 36% of original findings." (News coverage here.) While I'm sure many in the psychology field will seek to explain such findings, an inability to replicate 74% of studies published in "top" psychology journals is, at the very least, jarring.
Update 2: In a recent editorial in Psychology Science (here), D. Stephen Lindsay responds to a replication effort, published in Science (described above), that signaled serious problems. As Gelman (Columbia--Statistics) notes, "Lindsay talks about replication problems and how researchers should do better. He warns about p-hacking, noise, and the difference between significance and non-significance not being itself statistically significant." While it's notable to see a leading psychology journal recognize a problem and undertake concrete editorial policy changes to address the problem, it is likely only a (necessary) first step. To be sure, the "replication issue" is certainly not confined to psychology and it's an issue that warrants continued and sustained scholarly attention.
Update 3: From the 3.4.2016 NYT (here). Reporting on a paper critiquing the 2015 Science paper challenging replicability of 100 published psychology studies. Additional commentary from Gelman (Columbia--Statistics) here,here, and here.
Update 4: Marty Wells (Cornell--Statistics) notes that, according to a recent report in Science, replication problems have expanded into experimental economics.
In a recent paper, Dawn of the Discipline-Based Law Faculty, Lynn LoPucki (UCLA) documents the sharp increase in entry-level hires possessing PhDs at AALS-member law schools between 2011 and 2015. According to Stephen Bainbridge (UCLA), who "deplores" this trend, "LoPucki believes the trend in favor of PhD law professors has reached a point of no return, potentially cementing a future where law school faculty, are long on degrees and publications—but short on time outside university walls." Brian Leiter (Chicago) has also recently weighed in on this trend, and his take is a bit more measured and detached: "Although I'm quoted saying that the rise of JD/PhDs will continue, that's a descriptive not normative statement. I think different schools have different missions. And the relevance of the JD/PhD varies by field."
While the rise in empirical legal scholarship and the concurrent increase in law faculty possessing a JD/PhD combo are distinct, to ignore how these two trends interact is to blink at reality. As such, legal empiricists need to be mindful of these critiques.
UPDATE: Readers pointed me to another recent paper, The Ph.D. Rises in American Law Schools, 1960-2011: What Does It Mean for Legal Education?, by Justin McCray (Berkeley) et al. In it, the authors study faculty hiring at the 34th highest-ranked law schools between 1960-2011. After documenting a rise in PhD hiring, the authors note that "the shift toward Ph.D.s entails a complex set of benefits and costs for law schools, and that there is the potential for building connections between practical experience and academic research, rather than simply choosing between them."
Even the self-important newspaper of "record," the New York Times, could benefit from a basic statistics refresher. As Andrew Gelman (Columbia--Statistics) observed, a Feb. 11, 2016, NYT headline read as follows: "Gravitational Waves Detected, Confirming Einstein's Theory". Of course, as Gelman rightly points out, a more technically accurate headline would have read: "Gravitational Waves Detected, Not Falsifying Einstein's Theory".
While admittedly a slight departure--despite its timeliness--from traditional ELS Blog protocol, this post hopes to illustrate the "correlation v. causation" tension in a novel manner by drawing from, of all things, a recently-leaked Super Bowl 50 advertisement. The ad begins with an observation: "Data suggests 9 months after a Super Bowl victory, winning cities see a rise in births." The remaining wonderfully clever--and long--ad (here) then sets out to intentionally conflate correlation and causation. Regardless of one's interest level in football or Super Bowls, worth a look.
A recent PNAS article includes a figure illustrating “a marked increase in the all-cause mortality of middle-aged white non-Hispanic men and women in the United States between 1999 and 2013.” The data in the figure, however, were "not age-adjusted within the 10-y 45-54 age group.” Annual mortality rates were calculated by dividing the total number of deaths for the age group by the population of the age group. Also notable is that the NYT ran a story about this paper and prominently featured the (now misleading) figure (see below).
Enter Andrew Gelman (Coulmbia--Statistics) and Jonathan Auerbach (Columbia--Poli Sci) who correctly "suspected an aggregation bias and examined whether much of the increase in aggregate mortality rates for this age group could be due to the changing composition of the 45–54 year old age group over the 1990 to 2013 time period. If this were the case, the change in the group mortality rate over time may not reflect a change in age-specific mortality rates. Adjusting for age confirmed this suspicion. Contrary to Case and Deaton’s figure, we find there is no longer a steady increase in mortality rates for this age group. Instead there is an increasing trend from 1999–2005 and a constant trend thereafter. Moreover, stratifying age-adjusted mortality rates by sex shows a marked increase only for women and not men, contrary to the article’s headline" (emphasis added).
The important teaching take-away, as Gelman notes, is that "when performing reverse causal inference, remember that people move, and, as we’ve discussed before, the cohorts are changing. 45-54-year-olds in 1999 aren’t the same people as 45-54-year-olds in 2013. We adjust for changing age distributions (ya gotta do that) but we’re still talking about different cohorts."
As the Supreme Court struggles, once again, with diversity and affirmative action issues in the undergraduate admissions context in Fisher II (click here for the ScotusBlog summary), James Phillips (JD/PhD. cand. -- UC Berkeley) boldly and carefully assesses questions concerning ideological diversity in legal academia. Exploiting data on faculties at the 16 highest-ranked law schools, in Why are There so Few Conservatives and Libertarians in Legal Academia? An Empirical Exploration of Three Hypotheses Phillips engages with three standard explanations for the relative dearth of conservative law professors. The paper's abstract follows.
"There are few conservatives and libertarians in legal academia. Why? Three explanations are usually provided: the Brainpower, Interest, and Greed Hypotheses. Alternatively, it could be because of Discrimination. This paper explores these possibilities by looking at citation and publication rates by law professors at the 16 highest-ranked law schools in the country. Using regression analysis, propensity score matching, propensity score reweighting, nearest neighbor matching, and coarsened exact matching, this paper finds that after taking into account traditional correlates of scholarly ability, conservative and libertarian law professors are cited more and publish more than their peers. The paper also finds that they tend to have more of the traditional qualifications required of law professors than their peers, with a few exceptions. This paper indicates that, at least in the schools sampled, conservative and libertarian law professors are not few in number because of a lack of scholarly ability or professional qualifications. Further, the patterns do not prove, but are consistent with, a story of discrimination. The downsides to having so few conservatives and libertarians in the legal academy are also briefly explored."
Among the many aspects of the criminal justice system amenable to criticism, "life sentence without the possibility of parole" ("LWOP") has received particular attention recently, principally due to claims about its increased use. LWOP was conceived decades ago as a substitute for the death penalty. Consequently, while capital punishment wanes in the United States, a steadily increasing number of defendants is sentenced to LWOP. The surge in LWOP sentences has attracted critical attention. For example, a recent ACLU Report, critical of LWOP, asserts that "over 3,000 of the 50,000 inmates serving LWOP were convicted of nonviolent offenses."
"... In a few states, it is impossible to find a single inmate sentenced to LWOP for any crime other than murder or the most serious violent crimes. Even in jurisdictions that impose LWOP for crimes labeled “nonviolent,” the inmates are few in number and often present aggravating factors, such as extensive criminal histories or previous violent crimes. Inevitably, criminals sentenced to LWOP will vary in culpability, and some will appear not to merit this punishment. Drawing attention to their plight can spur executive clemency in individual cases. But accusations that the American legal system is rife with “ugly disproportionalities,” at least insofar as this claim is applied to LWOP sentences in the states, appear to have little merit.... Even in jurisdictions that impose LWOP for crimes labeled 'nonviolent,' the inmates are few in number and often present aggravating factors, such as extensive criminal histories or previous violent crimes."
Today's WSJ contributes reports on a recent paper (here) that seeks to contribute to a long-standing debate surrounding the proverbial "hot hand" fallacy. In their paper, Surprised by the Gambler's and Hot Hand Fallacies? A Truth in the Law of Small Numbers, Joshua Miller (Bocconi Univ.) and Adam Sanjurjo (Univ. de Alicante) suggest that a form of selection bias contributes to an underestimation of the true probability of an outcome when conditioned on a prior (and similar) outcome. Not sure if the paper works, but an intriguing potential wrinkle nonetheless.
While reasonable minds can (and do) differ on how best to legally operationalize the equal educational opportunity doctrine, if the doctrine is to stand for anything meaningful it has to at least mean that a child's educational "destiny" should be independent of the household income that a child happens to have been born into. Data, however, make painfully clear that on balance we fall short--far, far short--of this aspiration. Helping make these data "come alive" is an interactive graphic (in the New York Times) that allows users to "guess" at the relation between a child's chance at going to college and parental income levels. (After a user draws his/her own regression line onto the graph the user can then compare it with the actual regression line.) While depressing from a policy perspective it's a clever interactive graphic.
Empirical legal studies’ impact was on display last week in the Supreme Court’s death penalty decision, Glossip v. Gross. In particular, there is a lively back and forth among Breyer (joined by Ginsburg) in dissent, and the concurrences of Scalia and Thomas with no fewer than three JELS articles cited.
Suffice to say, Breyer and Ginsburg find the empirical evidence of problematic capital punishment application compelling, Thomas counters with questions about methods, and Scalia summarizes much of the scholarship as “abolitionist.” The link to the opinion is here. Cited article links follow.
The Supreme Court's recent Texas Department of Housing and Community Affairs v. The Inclusive Communities Project decision makes clear the broad point that the Fair Housing Act allows causes of action based on disparate impact claims rooted in statistical evidence. More specifically, however, the Court held that “[d]isparate-impact liability mandates only the removal of artificial, arbitrary, and unnecessary barriers” and that it is “not an instrument to force housing authorities to reorder their priorities.”
Consequently, what the decision means at a practical level remains anything but clear. Indeed, as William Consovoy observes at SCOTUSblog (here) "to defeat a disparate impact claim, housing authorities need only bring forth a 'valid interest served by their policies.' Furthermore, 'a disparate-impact claim that relies on a statistical disparity must fail if the plaintiff cannot point to a defendant’s policy or policies causing that disparity.' Indeed, the Court sent a strong signal that the disparate impact claim that gave rise to this dispute will have a difficult time moving forward under this standard."
This is a fascinating and complex topic where the rules/norms appear to be shifting in real-time. Some of the complexities are aptly illustrated in a recent post on Andrew Gelman's blog (here). (For those interested, even more valuable than the post itself is the growing list of comments that vividly illustrate this issue's potential complexities.)