Empirical Legal Studies

Northwestern University Law Review—3rd Annual Empirical Legal Scholarship Issue: Call for Papers

If evidence of empirical legal studies’ growth was not already obvious enough in the peer-reviewed world, traditional student law reviews also evidence a turn toward ELS as well. And no more so than perhaps the Northwestern University Law Review which is pleased to announce its third annual issue dedicated to empirical legal scholarship, to be published in Spring 2021. According to the student editors, “we welcome pieces making use of any and all empirical tools—including qualitative, quantitative, and mixed methods—to illuminate and engage questions of legal interest.”

“The Northwestern University Law Review accepts empirical articles on an exclusive basis only. The exclusive submission window for the Volume 115 Empirical Issue will open on February 15, 2020 and run until April 1, 2020. All authors will be informed by April 15, 2020 at the latest as to whether their piece has been selected to be sent out for peer review. At the final stage, authors will be informed no later than June 15, 2020 as to whether their piece has been selected for publication.”

“Interested authors must submit Articles and Essays via email to the Volume 115 Empirical Articles Editor at nulrempirical@gmail.com. Please submit the Article as a .doc or .docx file with a cover letter or similar email and CV." (For additional details click here.)

Law School Class Recordings and Student Access Policies: Data From Down Under

Recording classes and granting students liberal access to them are increasingly the norm in universities and law schools around the world. At the same time, however, mounting anecdotal evidence implies that increasing student access to class recordings corresponds with decreasing student class attendance. A recent paper, If You Record, They Will Not Come--But Does It Really Matter? Student Attendance and Lecture Recording at an Australian Law School, by Natalie Skead (Univ. Western Aus.) et al., presents and discusses results from a study that set out to test this “conventional wisdom” with a bit more rigor.

The paper is noted for its mixed methods which include a student survey, focus groups, and a manual count of observable student class attendance. Not only do the researchers' data speak to their specific research question (that is, the relation between student attendance rates and class recording access) but they also permit researchers to assess a derivative question—whether discrepancies exist between student attendance self-reports and actual observable student class attendance.

What the researchers found is that student access to class recordings is “strongly associated” with decreased student class attendance. As it relates to the reliability of self-reported survey data, the paper notes that the results “indicate a disparity between the high rates of self-reported attendance and the actual attendance count at lectures.”

Three quick closing points. First, while unlikely germane to the findings, it remains important to note that in Australia, and elsewhere (but unlike in the United States), law school includes both undergraduate as well as graduate students. Second, even assuming, arguendo, that the paper’s core findings hold, specific policy consequences relating to student access to class recordings are not obvious. Third, this research illustrates, yet again, why researchers need to approach self-reported survey data with caution given persistent reliability concerns.

Turning a Typical Stata Work Session Into A Do-File

Maybe others already knew this; I did not. As a multi-decade Stata user I've periodically had to transfer various past commands from my active window to a particular project's do-file. Doing so archives my analytical work on a project, helps me recall how I generated a specific set of results/tables/graphs, etc., and, concurrently, facilitates subsequent replication efforts (by me or anyone else). Until now to do so I've simply, and in an admittedly plodding-like and wholly inefficient manner, essentially hand "cut-and-paste" from the active window onto a separate do-file window. What I did not know--until a few hours ago--is that Stata has "buried" a way to accomplish all of this seamlessly and with a few mouse-clicks. This short Stata YouTube video (click here) shows how it's done.

CELS 2020--Dates/Location

Reflecting empirical legal studies' increasingly global footprint, I am delighted to announce that the Society for Empirical Legal Studies' 15th Annual Conference on Empirical Legal Studies (CELS) will be hosted by the University of Toronto Faculty of Law, take place on October 2-3, 2020, in Toronto, Canada, and co-organized by professors Albert Yoon, Anthony Niblett, and Andrew Green. More information on the call for papers will soon follow.

Simple Barcharts Options in Stata's Complex Graphics Coding

While I obviously should have stumbled onto this earlier, I'll continue to ascribe it to Stata's unnecessarily (in my view) complicated graphics coding. Indeed, complications hamstring even the "simple stuff."

The example below involves "style" elements incident to the standard, simple barchart. The conventional "graph hbar" command generates the "typical" barchart (on the left). The "statplot" command (user-written, requires downloading), however, generates (to my eyes, anyway) the slightly less "cluttered" and more "intuitive" barchart on the right. (Click here for additional information on "statplot".)

To be sure, this admittedly subtle distinction pivots on mere aesthetics and, of course, neither barchart chart is "wrong" in any way. They simply convey needed information in slightly different ways. But if I took anything from Tufte's classic exposition, The Visual Display of Quantitative Information, at some point "mere aesthetics" can matter. Or, if not "matter," can at the very least help or improve.

Legal Impact Research & Evidence of Racial Profiling

Far too many legal and court scholars (as well as law students, lawyers, judges, etc.) simply assume, at least as a presumptive matter, that court decisions yield the outcomes they seek. Not enough recognize how institutional arrangements, incentives, the "law of unintended consequences," etc., inform how court decisions are implemented "on the ground." A recent paper, An Empirical Assessment of Pretextual Stops and Racial Profiling, by Stephen Rushin (Loyola-Chicago) and Griffin Edwards (UAB-Economics), provides yet another helpful and timely reminder of the efficacy of subjecting formal legal doctrines to data and rigorous research designs.

In their paper the authors exploit a change in Washington State law concerning police traffic stops. While a 1999 Washington State Supreme Court decision (Ladson) barred pretextual traffic stops in 2012 the Washington Court reversed course (Arreola) by permitting "mixed motive" traffic stops. Varying this legal rule opened a door for a DID research design that draws on 8.2M traffic stops in Washington State between 2008 and 2015. And what the results suggest is that the Arreola court decision "was associated with a statistically significant increase in traffic stops and searches of nonwhite drivers relative to white drivers. We find that most of the increase in traffic stop of non-white drivers after Arreola occurred during the daytime, when police officers could more easily ascertain a driver’s race." Overall, this paper will interest not only because of the massive dataset but also for its empirical specification.

Standard (OLS) Regression for a *Binary* Dependant Variable?

Evidently, the question is not as crazy as I likely would have assumed 40, 30, or, perhaps, even 10 years ago. In a widely-discussed draft paper (click here), Logistic or linear? Estimating causal effects of treatments on binary outcomes using regression analysis, Robin Gomila (Princeton--(grad student) social psych) makes a case for OLS. While Andrew Gelman (Columbia--statistics) is among those who disagree (here), he also recommends alternative parallel suggestions and approaches. To the extent that the Gomila paper is partly motivated by the sometimes "muddled" interpretation of coefficients after a logistic transformation, my quick sense is that altering the empirical strategy risks a potential over-reaction.

New Goodness of Fit ("GOF") Test

Testing whether two separate distributions systematically vary typically involves a t-test or ksmirnov test. These two popular GOF tests involve comparing two distributions' means (t-test) or a hypothesized equal distribution (ksmirnov). A recently-released user-written command in Stata, distcomp, moves the needle a bit for GOF testing by assessing the equality of the distribution functions point-by-point. That is, instead of a single rejection (or non-rejection), the distcomp test generates ranges of values in which the two distributions' difference may achieve statistical significance.

To locate and download distcomp use the "findit" or "ssc install" functions in Stata. For a much more thorough explanation and description of the distcomp command click here.

Variable "Skew"

Any discussion about what to do with a "highly-skewed" variable needs to immediately differentiate between "highly-skewed" dependent and independent variables. A "highly-skewed" (or "non-normally distributed") dependent variable (or, more accurately, as it relates to the errors) raises all the obvious questions when it comes to the appropriateness of an OLS regression specification given its underlying core assumptions. A highly-skewed independent variable, by contrast, raises none of these particular concerns. As one of the commentators notes (see discussion, and an illustration, here), in certain contexts a highly-skewed independent variable may simply reflect "just the way the data are."

"Inflated" Standard Errors

While what constitutes an "inflated" standard error remains contested, what is clear, however, is that there is no standard, "go-to," statistical "fix" for any "inflated" standard errors. As one commentator notes (see StataList discussion here), "they are what they are." Rather, a more helpful approach towards "inflated" standard errors is to assess what in the underlying data set (or model) might be driving the standard errors. The standard set of "usual suspects" includes, e.g., outliers in the data set, an "extreme" level of multicollinearity, or an "over-fit" model. (For more on "over-fitting" regression models, see here.)

Outliers

While there are no obvious "hard-line" rules when it comes to identifying outliers in a data set--let alone what to do with any that are identified--a recent wide-ranging discussion is found here. Moreover, Professor Williams (Notre Dame--sociology) has put together an extremely helpful and rich resource document (here). Finally, another possibly useful listserv discussion (albeit a dated one) is found here.

Call For Papers: Conference on Empirical Legal Studies–Europe 2020

Evidencing the growing internationalization of empirical legal studies, the 3rd Conference on Empirical Legal Studies in Europe (CELSE) will be held at the University of Oslo, 11-12 June 2020.

"CELSE is an interdisciplinary gathering that draws researchers from across Europe and the world, bringing together scholars in law, economics, political science, psychology, policy analysis, and other fields who are interested in the empirical analysis of law and legal institutions. CELSE will consider empirical papers across all areas of law, although priority will be given to papers with a European focus. Empirical analysis is understood to encompass any systematic approach to quantitative or qualitative data analysis, including statistical analysis, machine learning, experiments, text-mining and network analysis techniques.

Papers are selected through a peer review process, and discussion at the conference includes assigned commentators and audience questions. Authors are encouraged to submit works-in-progress; however, submissions should be completed drafts that include principal results. Submitted papers must be unpublished (and expected to be unpublished at the time of the conference). If accepted, authors will have an opportunity to submit a revised draft prior to the conference for presentation and discussion. Please note that accepted papers will be made available to all conference participants."

The deadline for submissions is 15 February 2020. Anyone with additional questions or seeking further information should click here.

Judicial Citation Patterns and NY County Surrogate's Courts

As ELS scholars continue to develop judge citation studies--typically from the "top-down" (that is, we know comparatively far more about judge citation patterns in the SCOTUS and other federal courts than non-federal judicial contexts)--comparatively far less is know about judge citation patterns in various lower and state court systems. A recent paper by Bridget Crawford (Pace), What Probate Courts Cite: Lessons from the New York County Surrogate's Court 2017-2018, contributes to a scholarly gap by describing the basic contours of judge citation patterns in New York County Surrogate's Court during 2017 and 2018. New York County's Surrogate's courts serve the most densely populated county in the United States. An excerpted abstract follows.

"... This Article makes three principal claims — one empirical, one interpretative, and one normative. This Article demonstrates through data, derived from a study of all decrees and orders issued by the New York County Surrogate’s Court in the years 2017 and 2018, that the probate court located in the most densely populated county in the United States cites fewer authorities less often than almost any other court (of any level) for which data is available.

There are a variety of factors that may explain this low rate of citation by the New York County Surrogate’s Court including docket size, the size and composition of the court’s staff, a judicial perception that the application of the law is a relatively mechanical process, or a subjective determination that speed in processing the court’s docket outweighs any public interest in citation-replete decrees and orders. Yet by increasing its engagement with a range of authorities, the New York County Surrogate’s Court (and indeed any probate court) may increase public confidence in the judiciary while also enhancing understanding of trusts and estates as a complex and dynamic area of law."

NYT, MTurk, and Data

Yesterday's New York Times ran a lengthy piece (here) on Amazon's Mechanical Turk ("MTurk") platform. The author joined the ranks of "Turkers" for a brief period and did some "Turking" to supplement his background research for the article. While the piece reflects a decidedly labor-economic focus, insofar as "MTurk"-created data remain a staple for an increasing slice of experimental psychological research (despite earlier concerns raised years ago by, among others, Dan Kahan (see, e.g., here and here)), the NYT''s article warrants a quick read.

CELS 2019--Conference Schedule

Co-sponsored by Claremont McKenna College and the Society for Empirical Legal Studies (SELS), the 14th annual CELS 2019 conference schedule is here.