Empirical Legal Studies

MTurk and Evidence on Measurement Error

While questions surrounding the potential for systematic measurement error from MTurk samples cannot be easily resolved in a single paper, given the popularity of MTurk samples in many fields, notably psychology, careful attention scholarly is warranted. The initial "warning" about MTurk samples came in 2013 from Dan Kahan (Yale), particularly for studies seeking to test “hypotheses about cognition and political conflict over societal risks and other policy-relevant facts.” Since then we've seen a developing literature that sets out to test potential limitations to MTurk samples. A recent paper by Douglas Ahler (FSU-Poli Sci ) et al., The Micro-Task Market for Lemons: Data Quality on Amazon’s Mechanical Turk, focuses on the potential for measurement error, especially Type II errors, injected by, among other factors, the "presence of 'non-respondents' (bots) or non-serious respondents on the platform." The paper's core findings include:

"While we find no evidence of a 'bot epidemic,' we do find that a significant portion of survey respondents engaged in suspicious behavior. About 20% of respondents either circumvented location requirements or took the survey multiple times. In addition, at least 5-7% of participants likely engaged in 'trolling' or satisficing. Altogether, we find about a quarter of data collected on MTurk is potentially untrustworthy. Expectedly, we find response quality impacts experimental treatments. On average, low quality responses attenuate treatment effects by approximately 9%. We conclude by providing recommendations for collecting data on MTurk."

Cognitive Bias, Judges, and Evidence Admissibility Contests

While research on individuals' susceptibility to various cognitive biases has blossomed over the years, the application of this particular research to judges and their decisions is less well-developed. A recent paper by Jeffrey Segal et al. (Stony Brook--Poli Sci), The 'Murder Scene Exception'—Myth or Reality? Empirically Testing the Influence of Crime Severity in Federal Search-and-Seizure Cases, contributes to the literature by assessing whether variation in crime severity informs judges' resolutions of contests over evidence admission. The paper draws from a sample (N=495) of federal appellate search-and-seizure cases from 1972 to 1986 and the excerpted abstract (below) summarizes its core findings.

"Using legislative criminal penalties to measure crime severity, our analysis shows that as crime severity increases, judges become significantly less likely to exclude challenged evidence on Fourth Amendment grounds—even though crime severity is not a doctrinally relevant consideration. Another legally extrinsic factor, the ideology of the opinion-writing judge, is also found to exert an influence, but only in the most serious criminal cases that involve a life sentence or the death penalty. In these particularly high-stakes decisions, conservative-leaning judges are more likely to uphold the admissibility of challenged evidence, while liberal-leaning judges are more likely to suppress it. Our data also indicate that the intrusiveness of the challenged police search, a doctrinally relevant factor, independently influences admissibility judgments."

When--and How--To Use Kruskal-Wallis

Helpful discussion, and example, of how to execute the Kruskal-Wallis ("K-W") test as well as an explanation on why one would turn to it (click here). In general (and in brief), folks typically turn to the K-W test when you have one nominal variable (e.g., gender) and one measurement variable (e.g., test scores). While one-way anova is used when the measurement variable distributes normally, K-W is used when the measurement variable does not.

Balanced v. Unbalanced Panel Data; "reg" v. "xtreg" Commands--Asking the Right Question

An admittedly small but nonetheless important point emerged in a recent StataList discussion (here). While the original query wondered whether a decision between the "reg" and "xtreg" commands pivoted on whether panel data were balanced or unbalanced, a very helpful commenter quickly made clear the question's pivotal (and mistaken) assumption. To be sure, while the "reg" and "xtreg" commands may yield the same or similar results in certain conditions, they estimate different models and different parameters. In general, where panel data are involved (balanced or unbalanced), as the commenter notes,"-reg- is simply inappropriate."

CELS 2019 Submission Deadline--Extended to June 28, 2019 (Midnight, Pacific Time)

Claremont McKenna College, led by conference co-organizers Eric Helland, David Bjerk, and Daniel Krauss, and the Society for Empirical Legal Studies (SELS) are pleased to announce the 14th Annual Conference on Empirical Legal Studies (CELS) on November 15-16, 2019, in Claremont, California.

CELS is a highly regarded interdisciplinary gathering that draws researchers from across the United States and the world, bringing together scholars in law, economics, political science, psychology, policy analysis, and other fields who are interested in the empirical analysis of law and legal institutions. Papers are selected through a peer review process, and discussion at the conference includes assigned commentators and audience questions.

CELS will consider empirical papers spanning all areas of empirical legal studies. Authors are encouraged to submit works-in-progress; however, submissions should be completed drafts that include principal results. Submitted papers must be unpublished (and expected to be unpublished at the time of the conference). If accepted, authors will have an opportunity to submit a revised draft prior to the conference for presentation and discussion. Please note that accepted papers will be made available to all conference participants.

The paper submission deadline has been extended to June 28, 2019 (midnight, Pacific Time). To submit a paper or for more information about CELS 2019, including conference registration, please go to the CELS 2019 conference website (here).