As a co-editor I note with pride that JELS 10:3 maintains (for more than one decade) JELS' perfect record of on-time publication and includes a wonderful collection of diverse and interesting papers. Topics in this issue range from data on the Indian Supreme Court's workload (here) to multidistrict litigation transfers and consolidations (here).
Prompted by my prior post discussing statistical significance levels, my Cornell colleague Ted Eisenberg passed along this 1982 paper from the American Psychologist by Michael Cowles (York) and Caroline Davis (York) discussing the emergence of the p < 0.05 threshold as the "standard" in the social sciences. Cowles and Davis argue that the move to the p < 0.05 threshold pre-dates Sir Ronald Fisher's contribution.
Rafael Irizarry makes an interesting contribution to a discussion about whether the standard statistical significance threshold (p < 0.05) should be tightened up.
"The gist of my thought is that, for some scientific fields, the
pessimist's criticism is missing a critical point: that in practice,
there is an inverse relationship between increasing rates of true
discoveries and decreasing rates of false discoveries and that true
discoveries from fields such as the biomedical sciences provide an
enormous benefit to society."
While Irizarry's comment dwells on the hard sciences, recognizing the
inevitable trade off between stringency and discovery (by plotting true
positive and false positive rates for a given classifying procedure)
potentially implicates empirical legal scholarship as well.
From the archives comes a helpful reminder from Osborne & Waters about key regression assumptions that require examination (here). Gelman provides a "friendly" amendment and a (related) list of his own.
To prolong my recent mini-obsession on statistical power issues, I thought some readers might find a three-part statistical power "webinar" helpful. Hosted by UCLA's outstanding Institute for Digital Research and Education (idre), the power analysis series includes introductory, intermediate, and advanced instruction. Insofar as power issues are too often ignored by researchers, this resource is worth a look.
David Schwartz (Chi-Kent) asked that I pass along on the inaugural "Workshop on Empirical Methods in Intellectual Property," co-sponsored by Chicago-Kent's Center for Empirical Studies of Intellectual Property and the U.S. Patent and Trademark Office. I am happy to oblige and welcome another field-specific empirical methods workshop into the fold. Summary information follows.
"The workshop is intended to give scholars engaging in empirical and experimental studies of IP a chance to receive feedback on their work at an early stage in their research. Accordingly, the workshop will be limited to a small cohort of scholars discussing projects that are still in their developmental stages. Projects that will have substantially begun data collection by the time of the workshop are inappropriate. Pilot data collection is, however, appropriate."
"The workshop will be organized around a modest number of projects. Each project presenter will be expected to circulate a description of the project of no more than 10 pages by October 1. Each project will be assigned to an expert commenter and will be allotted 45 minutes of discussion by the attendees."
For those interested, applications are due by August 24. Decisions will be made by September 2. Click here for detailed information; email Christi Guerri (email@example.com) with any questions.
A series of posts (one dating back to 2004) by Andrew Gelman (Columbia--Statistics), including links to helpful papers, provides a nice overview of the array of potential errors flowing from under-powered papers. A refresher on the nomenclature (here) sets forth the basics.
"A Type 1 error is commtted if we reject the null hypothesis when it is true. A Type 2 error is committed if we accept the null hypothesis when it is false. A Type S error is an error of sign. A Type M error is an error of magnitude."
Later posts (here), along with relevant papers (here and here), develop the issue further. Insofar as the number of papers I see that neglect reporting results from basic power tests remains far too large, these links might interest some readers.
Paul Collins, Jr. (Univ. N. Texas--Poli Sci) notes the public availability of the U.S. Supreme Court Confirmation Hearings Database that he and co-author Lori Ringhand used in their recent book, Supreme Court Confirmation Hearings and
Constitutional Change (Cambridge Press, 2013).
"This database provides a wealth of information regarding the confirmation
hearings of U.S. Supreme Court nominees held before the Senate Judiciary
Committee. Based on confirmation hearing dialogue, the dataset includes
information on the political environment surrounding the nomination, the issue
and subissue areas being discussed, and the manner in which the nominees answer
senators' questions. In addition, the database contains information on the
discussion of judicial decisions at the hearings, including the name of the
decisions and the courts that rendered the cases debated at the hearings."
visit the site (here) will note that it includes a growing
compilation of data sets--most germane to federal court judicial decisionmaking--linked
to selected publications that exploit the data. Also, relevant Stata do files accompany many of the data sets. Not only does this site help disseminate useful data, but it also facilitates replication efforts.
Two final notes. First, when JELS editors (myself included) or JELS referees request data and do files from authors incident to manuscript reviews, Paul's web site includes examples of "best practices" that should be widely emulated. Second, graduate and law students seeking to learn empirical methods will find these "ready-to-use" data sets invaluable.
Many folks, especially those who conduct psychology experiments, will want to consider carefully Dan Kahan's (Yale) recent post about problems with relying on Mechanical Turk for data. More specifically, Kahan discusses "the invalidity of studies that use samples of Mechanical Turk workers to
test hypotheses about cognition and political conflict over societal
risks and other policy-relevant facts."
According to Dan, the "three decisive 'sample validity' problems" include: selection bias, prior, repeated exposure to study measures, and subjects' nationality misrepresentation. These problems, according to Dan, render Mechanical Turk samples particularly problematic for studies of culturally or
ideologically grounded forms of “motivated reasoning.”
I am delighted to note that Stanford Law Review's current issue (65:6) focuses exclusively on empirical legal studies. The issue features essays from six Stanford faculty members who used papers presented at last year's CELS (hosted by Stanford Law School) as a starting point to discuss the state of empirical work in an array of substantive legal sub-fields. The introductory essay, The Empirical Revolution in Law, by Prof. Dan Ho (Stanford) and Stanford Law's former Dean, Larry Kramer, provides a nice frame and overview. I recommend the entire issue to all.
I should also note that the 2013 CELS, hosted by Penn Law School, is scheduled for October 25-26, 2013. Those hoping to present papers or posters need to submit by no later than midnight (EST) on Friday, July 12, 2013.