Empirical Legal Studies

More Evidence of Problems With Survey Data

For those who use survey data, the Brexit outcome renews long-standing concerns. Andrew Gelman (Columbia--Statistics) ascribes the difference between most leading surveys (polls) and the Breixt election outcome to five factors (here). Of the five that Andrew identifies, the most troubling (and problematic) is No.2 (what Andrew labels--charitably--as "measurement error"). To be clear, what is implied by "measurement error" in the survey context is that what respondents say they do and actually do (observational data) can vary. To put the point more crassly, some (unknown percentage of) respondents either lie or "delude" themselves. While measurement error always persists as an important threat to data, this threat is particularly acute for survey data. While what to do about this threat isn't obvious, simply ignoring it not a plausible option in most contexts.

ELS & Law Reviews: Possible Ways Forward

Despite the continued growth of faculty-edited, peer-reviewed journals for empirical legal scholars, traditional student-edited law reviews continue to receive and review a growing number of empirical paper submissions. When it comes to reviewing empirical work, however, substantial limitations baked into law reviews' structure pose important challenges. These limitations include the annual turnover of student editors most of whom lack the technical and subject-area expertise necessary to assess sophisticated empirical work. How to improve this situation, while honoring student law reviews' traditional autonomy and pedagogical goals, persists as an important challenge. In her recent paper, The Future of Empirical Legal Scholarship: Where Might We Go From Here?, Kathy Zeiler (Boston Univ.) approaches this challenge from an incentives perspective. The abstract follows.

“The number of empirical legal studies published by academic journals is on the rise. Given theory’s dominance over the last few decades, this is a welcomed development. This movement, however, has been plagued by a lack of rigor and a failure of editors to require disclosure of data and procedures that allow for easy replication of published results. Law journals, the editorial boards of which are manned solely by law students, might face the toughest hurdles in ensuring publication of only high quality empirical studies and in implementing and enforcing disclosure policies. While scholars in other fields including economics, psychology, and political science seem to be taking steps to address widespread quality issues, little is being done to address the problems in the law literature. The purpose of this essay is to argue that most proposed solutions offered over the last decade or so have not taken hold because they do not generate incentives powered by the interests of actors in positions to affect change. Those that have caught on might be causing more harm than good. I offer a set of proposals grounded in a framework based on tapping into the interests of both law journal student-editors and authors who submit empirical legal studies to law journals for publication. Improving the quality of empirical studies will require time and sustained effort, but the hope is that successful implementation of them (or others that provide strong incentives for change) will reduce the need for effort over time.”

From the Perspective of Editors and Reviewers

While this particular paper and related post have little to do with ELS per se, Gelman's observations along with the post's comments may help authors understand how editors and reviewers discharge their functions. The paper's small sample (N=44) almost dooms it from the start. But, as the comments make clear, other problems emerge as well. Bottom line, if a paper's sample, methodology, research design, or analysis, etc., don't pass a "smell test" (basic persuasiveness which pivots to some degree on one's sense of replicability), most editors/journals will pass on it. (That the paper achieved publication--in The Quarterly Journal of Experimental Psychology--was itself insufficient to insulate it from important criticism.)

Reversing Scale of a 7-Point Likert Variable -- Quick Coding Tip

A graduate student recently posed the following question (edited) to the Stata List which prompted a very helpful, quick reply:

Q: My data include numerous 7-point Likert-scale items and I need to recode some due to inconsistent question directionality (e.g., some variables are worded "I like working here," while others "I would take an other job if i could"). Thus, I need to "re-scale" some variables to ensure consistent interpretation.

A: This a matter of reversing a scale. Some might want to point you to recode here but I would want to underline that.

Code:

gen myvar2 = 8 - myvar

This flips 1 to 7 and vice versa and correspondingly flips in between. Value labels will need fixing.

Obviously, coding necessary to "flip" values for a 5-point Likert scale would involve the number "6", etc.