Empirical Legal Studies

Testing Police Officers' "Intuition" About "High-Crime" Areas

In a fascinating paper, The End of Intuition-Based High-Crime Areas, Ben Grunwald (Duke) and Jeffrey Fagan (Columbia) engage an important criminal procedure question with the benefit of a massive data set. The paper begins with the Supreme Court's conclusion in Wardlow that a "suspect’s presence in a 'high-crime area' is relevant in determining whether an officer has reasonable suspicion to conduct an investigative stop." Perhaps not surprisingly, the Court's opinion in 2000 provides little guidance about what constitutes a "high-crime" area. The paper sets out to assess how the doctrine was operationalized by examining data on more than two million investigative stops conducted by the NYC police from 2007 to 2012.

The data are especially rich in that in NYC police officers who conduct an investigative stop are required to complete a form after each one. "The forms collect rich information on suspect demographics and the precise geographic location of each stop. The data also contain anonymized officer identifiers, which allow us to observe how the same officer behaves in different areas, and how different officers behave in the same areas. And, most important for our purposes, the forms require officers to check off a series of roughly twenty boxes, indicating the bases of suspicion that justified the stop. Fortunately, one of those boxes is for high-crime areas. We merged this dataset with crime statistics and racial and demographic information on small geographic areas in New York City.”

As the paper makes clear, the data cast uneasy light on the Wardlow decision. "Our results suggest that Wardlow may have been wrongly decided. Specifically, we find evidence that officers often assess whether areas are high crime using a very broad geographic lens; that they call almost every block in the city high crime; that their assessments of whether an area is high crime are nearly uncorrelated with actual crime rates; that the suspect’s race predicts whether an officer calls an area high crime as well as the actual crime rate; that the racial composition of the area and the identity of the officer are stronger predictors of whether an officer calls an area high crime than the crime rate itself; and that stops are less or as likely to result in the detection of contraband when an officer invokes high-crime area as a basis of a stop. We conclude with several policy proposals for courts, police departments, and scholars to help address these problems in the doctrine.”

Equally interesting is that the authors point the way for fruitful follow-up empirical work on this topic. "In future empirical work, legal scholars can use body camera footage or SSO to validate the most common factors invoked by officers to establish reasonable suspicion. They can do so by assessing the factual accuracy of police officers’ claims about the presence of these factors and by examining whether they are in fact predictive of criminal behavior. Armed with that information, scholars and courts will have a clearer picture of what Fourth Amendment factors provide a reliable basis for reasonable suspicion and meaningful protection against unreasonable intrusions on personal privacy."

A New Legal Cloud Over Statistical Analyses of Judicial Decisions in France

In a move that will strike many court scholars (and, likely, others as well) as verging on the "unbelievable" (in the literal sense of the word), according to emerging news reports (e.g, here) the French Government has now "banned the publication of statistical information about judges’ decisions–with a five year prison sentence set as the maximum punishment for anyone who breaks the new law." The new french law (Art. 33 of France's Justice Reform Act) makes clear that "[t]he identity data of magistrates and members of the judiciary cannot be reused with the purpose or effect of evaluating, analysing, comparing or predicting their actual or alleged professional practices.” (translated; emphasis added)

According to news accounts, the new french law targets "legal tech companies focused on litigation prediction and analytics–from publicly revealing the pattern of judges’ behaviour in relation to court decisions." That said, the implications for empirical legal scholars seeking to model judicial decisions, certainly at the individual judge-level, in France are obvious.

While empirical legal scholars seeking to analyze U.S. judicial decisions face bothersome "hurdles" when it comes to linking judge names and their published decisions (e.g., a long-standing AO policy that strips judge names/IDs from most publicly-available AO data sets), I am unaware of anything in the U.S. that comes close to what the French Government seeks to accomplish with its new law.

Data on Immigration Judge Decisions Over Time

Public (and scholarly) debate over immigration laws and their enforcement all too often takes place in a data vacuum. A recent paper by Catherine Kim (UNC) and Amy Semet (Columbia), An Empirical Study of Political Control Over Immigration Adjudication, sets out to reduce this data paucity. Exploiting a large data set that includes every immigration removal proceeding decision from January 2001 through November 2018 (N=800,000+), the paper endeavors to isolate and assess the potential influence of an Administration's political preferences. The excerpted abstract, below, summarizes the paper's key findings.

"This Article presents the first comprehensive empirical assessment of the extent to which Immigration Judges (IJs), the administrative officials charged with adjudicating whether a given noncitizen will be deported from the country, decide cases on the basis of an Administration’s political preferences rather than an independent assessment of the legal merits of a given case. 

We constructed an original dataset of over 800,000 removal proceedings decided between January 2001 and November 2018. First, we found that every Administration—not just the current one—disproportionately appoints IJs with backgrounds in the former Immigration and Naturalization Service (INS), the Department of Homeland Security (DHS), or the Department of Justice (DOJ), agencies with responsibilities in prosecuting noncitizens. Second, using logistical regression to control for over a dozen variables that might impact a decision to remove, we found that the identity of the Administration that appointed an IJ was not a statistically significant predictor of the likelihood of ordering removal. That is, after controlling for other variables, Trump appointees were no more likely to order removal than Obama or Bush appointees.

Finally, using logistical regression and controlling for other variables, we found that the Administration in control at the time of decision is a statistically significant predictor of removal rates. For example, Bush appointees who served during the Bush, Obama, and Trump Eras were far more likely to order removal during the Trump Era. Specifically, when all other variables are held constant, they were 14% less likely to order removal during the Obama Era than during the current Era, and 12% less likely to order removal during the Bush Era than during the Trump Era. These results suggest that the current Executive-in-Chief exercises a profound influence over removal decisions, undermining the assumption of independence among administrative adjudicators."

Tips for More Readable Graphs

Over at Andrew Gelman's blog, John Kastellec (Princeton--Poli Sci) ignited a discussion (here) about "common but easily avoidable graphical mistakes" that hinder graphs' readability. While some suggestions are obvious, others are not. As well, the comments to John's post are as important and helpful as the post itself. When all is said and done, however, for my money the most important contribution to this topic remains Tufte's classic: The Visual Display of Quantitative Information (2nd ed.).

"georoute"--Measuring Distance and Time Between Two Points

While requiring a user-download, a new--albeit quirky--command, "georoute", allows researchers to compute "travel distance and travel time between two points defined by their addresses or their geographical coordinates." Or, more specifically:

"calculates the georouting distance between two addresses or two geographical points identified by their coordinates. It uses the HERE API (https://developer.here.com) to retrieve distances in two steps. In the first step, addresses are geocoded and their geographical coordinates (latitude and longitude) are obtained. In the second step, the georouting distance between the two points is obtained. The user can also provide directly geographical coordinates, which will bypass the first step."

Such a program may prove useful, e.g., to researchers exploring various "access to justice" research papers that seek to exploit/explore data on physical distance. A brief discussion is found here.

For those interested, to download this program, from within Stata, type: ssc install georoute

Incorporation & the "Delaware Trap"

In The Delaware Trap: An Empirical Analysis of Incorporation Decisions, Robert Anderson (Pepperdine) engages with, as he notes, "One of the most significant debates in corporate law," namely "whether the United States system of corporate law federalism leads to a race to the bottom or a race to the top. Race-to-the-bottom theorists argue that because insiders of companies must initiate incorporation decisions, jurisdictions compete to provide legal rules that favor insiders, allowing them to extract private benefits at the expense of the corporation or its shareholders. Race to-the-top theorists argue that market constraints prevent insiders from favoring such jurisdictions and that jurisdictions actually compete to provide efficient legal rules that enhance shareholder value. Although the dichotomous framing as a 'race' to the 'top' or 'bottom' is a bit of an oversimplification of a more nuanced debate, that version of the debate has dominated discussions of corporate law for decades.” And to this classic research question Anderson brings a fresh set of under-explored data: "filings made with the Securities and Exchange Commission under Regulation D. Regulation D is probably the most commonly used exemption from the Securities Act of 1933; it is used for private capital-raising and is relied on by over 10,000 public and private companies each year.”

Anderson's findings emphasize "“demographic markers of sophistication, such as choice of law firm and headquarters location, predict the jurisdictional choice about as well as state law or the business attributes of companies. Companies with more demographic markers of sophistication tend to choose Delaware incorporation, and companies with fewer demographic markers of sophistication tend to choose home-state incorporation.” And these findings persist after including many of the obvious controls.

Anderson interprets these results to imply that "the choice of legal representation is an important missing variable in models of incorporation decisions. The fact that the choice of law firms drives the jurisdictional choice has far broader implications. If law firms drive the jurisdictional choice they may steer companies toward states that serve the law firms’ own interests without regard to the quality of legal rules or the needs of the client. When the state chosen is Delaware, as it often is, there are few alternative jurisdictions that shareholders and managers can agree on. As a result, companies inadvertently fall into a 'governance trap' from which reincorporation out of state is nearly impossible. This interpretation would suggest that Delaware’s carefully calibrated positioning in the charter market has largely eliminated meaningful competition among the states for the quality of corporate law."

Prof. Frank R. Cross (1955-2019)

It is with sadness and a widely-shared sense of loss that I pass along the following note (from Prof. Stephen Griffin) on Frank Cross' passing.

"I wanted to record the passing of my wonderful friend Frank Cross, a longtime faculty member at the University of Texas, Austin. Other obituaries will recall that he was one of the best (NDT) college debaters of the 1970s and a gifted teacher. I wish to highlight the tremendous loss to academia and legal scholarship. Frank contributed enormously by continually challenging himself. Already a UT faculty member, he embarked on a course of study in statistics that transformed his work and gave us new insights. He began an extraordinary run of publication in leading law reviews with an article that questioned work that downplayed the contribution of lawyers to the economy. He published many noteworthy books including Decision Making in the U.S. Courts of Appeals (2007); Measuring Judicial Activism (2009)(with Stefanie A. Lindquist); The Failed Promise of Originalism (2012); The Theory and Practice of Statutory Interpretation (2012); and Constitutions and Religious Freedom (2015). Frank contributed as long as he could. Robert Prentice, his friend on the UT faculty, said in his obituary: 'Frank bravely battled adrenoleukodystrophy his entire adult life. This cruel disease slowly robbed Frank of his ability to move, but never took away his love of life or sense of humor. ALD ultimately defeated Frank, but it did not define him.' Everyone who knew Frank will miss him. He had this rare gift: an independent mind and an independent heart."

Model Options For Log-Transformed Dependent Variables

This month's Stata News (34:2) includes a quite helpful (and instructional--with examples and coding) discussion about some classic challenges that arise from a log-transformed dependent variable. First, the need to log-transform dependent variables is exceedingly common in ELS (e.g., punitive damages models). Second, when log transformations are necessary interpretation challenges arise as few among us think in terms of, e.g., "log punitive damages" or "log wages." The essay walks readers through various model options (regression and poisson) and how to exploit the margins command to assist with interpreting results even further.

How to Deal with an Ordinal Dep Var When You Need/Want to Include Fixed Effects?

A recent question posed on StataList has stimulated an interesting discussion (here) as well as a call for FE ordinal models. The specific question:

"So to sum up my question: Is it possible to 'argue' that my ordinal dependent variables can be treated as continuous and I can therefore use FE for them as well? If yes - could you point me towards the appropriate literature/methodology? Or is this not possible and I need to use logit/probit for the ordinal dependent variables?"

The core of the question--can an ordinal dependent variable be treated as "continuous"--comes up most frequently in the survey context where a response involves a Likert Scale (typically a 5- or 7-point scale -- or an even-numbered scale that pushes respondents off a mid-point response). While reasonable minds can--and do--differ on what modeling options may plausibly exist, at the very least, and as a first step, a researcher would need to closely examine how any ordinal dependent variable distributes as this may itself limit model selection options.

Northwestern University Law Review--2nd Annual Empirical Legal Scholarship Issue: Call for Papers

The Northwestern University Law Review is pleased to announce its second annual issue dedicated to empirical legal scholarship, to be published in Spring 2020. We welcome pieces making use of any and all empirical tools—including qualitative, quantitative, and mixed methods—to illuminate and engage questions of legal interest.

The Northwestern University Law Review accepts empirical articles on an exclusive basis only. The exclusive submission window for the 2020 empirical issue will run from March 15–April 15, 2019. Participating authors must agree to withhold the manuscript from submission to any other publications until receiving a decision from us. All publication decisions will be released no later than July 31, 2019.

Interested authors must submit articles and essays via email to Empirical Articles Editor Kerri Howard (at: kerri.howard@nlaw.northwestern.edu).

Please submit the article as a .doc or .docx file with a cover letter or similar email; CV; and, if desired, supporting materials. More information about submission requirements and the empirical selection process is available here.

CELS 2019 -- Call For Papers

Spearheaded by CELS 2019 co-presidents David Bjerk, Eric Helland, and Dan Krauss, Claremont McKenna College and the Society for Empirical Legal Studies (SELS) are pleased to announce the 14th Annual Conference on Empirical Legal Studies (CELS) on November 15-16, 2019, in Claremont, California.

CELS is a highly regarded interdisciplinary gathering that draws researchers from across the United States and the world, bringing together scholars in law, economics, political science, psychology, policy analysis, and other fields who are interested in the empirical analysis of law and legal institutions. Papers are selected through a peer review process, and discussion at the conference includes assigned commentators and audience questions.

CELS will consider empirical papers spanning all areas of empirical legal studies. Authors are encouraged to submit works-in-progress; however, submissions should be completed drafts that include principal results. Submitted papers must be unpublished (and expected to be unpublished at the time of the conference). If accepted, authors will have an opportunity to submit a revised draft prior to the conference for presentation and discussion. Please note that accepted papers will be made available to all conference participants.

For general information about the CELS 2019 conference (click here); to submit a paper (click here).  Please note the paper submission deadline: June 21, 2019 (midnight, Pacific Time).

"Significance" in Marginal Effects v. "Original" Model

A lively discussion (here) on the Stata list takes up the question: "This variable has significant effects in the original (logit/probit/heckprobit/whatever) model. But its marginal effect is not significant. Why?"

As the ensuing (and on-going) discussion makes clear, this nuanced discussion involves, among other things, careful attention to precisely what specific hypotheses are being tested in a margins specification versus an "original" specification.

Data on Post-Conviction Criminal Expungement

As briefly described in a recent NYT op-ed (on March 20, 2019; link here for those who can get behind the NYT pay-wall), Professors J.J. Prescott (Mich.) and Sonja Starr (Mich.) submit an increasingly discussed policy--a push for increased expungements of criminal convictions--to data. As the paper notes, motivating the increased policy attention to expungements are concerns about "reentry barriers faced by people with criminal records." Contributing to an under-developed empirical literature on the possible impacts of criminal expungements is a paucity of public information on expunged criminal records. In Expungement of Criminal Convictions: An Empirical Study, Prescott and Starr lever their access to "complete, deidentified criminal records from the Michigan State Police on all individuals who had obtained criminal record “set-asides” (Michigan’s term for record-sealing) as of March 2014, as well as full criminal history records for much larger comparison groups of individuals with convictions that were not set aside."

The paper's careful attention to research design addresses one (but not all) traditional threat to identification posed by selection bias. By adopting a within-person design--and not comparisons between those received expungements and those who did not--the specification's identification is not distorted by the likely (almost assured) systematic differences between expungee and non-expungee sub-groups. That said, even the paper's within group design can not fully blunt potential distortions owing to variations in the timing of an individuals' expungement application. A summary of key findings is included in the abstract.

"Laws permitting the expungement of criminal convictions are a key component of modern criminal justice reform efforts and have been the subject of a recent upsurge of legislative activity. This debate has been almost entirely devoid of evidence about the laws’ effects, in part because the necessary data (such as sealed records themselves) have been unavailable. We were able to obtain access to deidentified data that overcomes that problem, and we use it to carry out a comprehensive statewide study of expungement recipients and comparable non-recipients. We offer three key sets of empirical findings. First, among those legally eligible for expungement, just 6.5% obtain it within five years of eligibility. Drawing on patterns in our data as well as interviews with expungement lawyers, we point to reasons for this serious “uptake gap.” Second, those who do obtain expungement have extremely low subsequent crime rates, comparing favorably to the general population—a finding that defuses a common public-safety objection to expungement laws. Third, those who obtain expungement experience a sharp upturn in their wage and employment trajectories; on average, within two years, wages go up by 25% versus the pre-expungement trajectory, an effect mostly driven by unemployed people finding jobs and very minimally employed people finding steadier or higher-paying work."

SCOTUS, Flowers v. Mississippi, Social Science & Cornell Law School News

My CLS colleague, Sheri Johnson, is scheduled to argue Flowers v. Mississippi in the Supreme Court tomorrow. In addition to falling into Sheri’s scholarly (and lawyerly) sweet-spot, post-conviction capital defense appeals, this particular case (and appeal) has generated significant public attention flowing in part from a recent In The Dark podcast episode (Season 2: Curtis Flowers; click here). Tomorrow’s argument will engage with issues related to the prosecutor's peremptory challenges and possible Batson problems.

And were that not enough, another CLS colleague, Valerie Hans, invested some time in her Social Science in the Law course this semester on an array of underlying complex social science and legal dimensions raised by the Flowers litigation. (For a flavor of the issues covered click here.)

2019 Northwestern-Duke Causal Inference Workshops

Northwestern and Duke law schools, along with the Society for Empirical Legal Studies, will co-sponsor the 10th annual Research Design for Causal Inference workshops. Co-organized by Bernie Black (Northwestern) and Matthew McCubbins (Duke) and hosted at Duke University School of Law, in Raleigh-Durham, NC, the Main workshop is set for August 12-16, 2019, with an Advanced workshop following on August 19-21, 2019 (also at Duke). As the workshop overview notes, "Research design for causal inference is at the heart of a 'credibility revolution' in empirical research. We will cover the design of true randomized experiments and contrast them to natural or quasi experiments and to pure observational studies, where part of the sample is treated in some way, the remainder is a control group, but the researcher controls neither the assignment of cases to treatment and control groups nor administration of the treatment. We will assess the causal inferences one can draw from a research design, threats to valid inference, and research designs that can mitigate those threats." To these ends, the Main workshop will cover the basics of causal inference, including potential outcomes, matching; difference-in-differences, instrumental variables, and regression discontinuity designs. The Advanced workshop will focus on principal stratification (a generalization of causal IV analysis), advanced matching with panel data, and application of machine learning to causal inference. Both workshops feature an outstanding faculty and provide an excellent and efficient way to become acquainted with contemporary approaches for making causal inferences from various kinds of observational and experimental data. For those seeking additional registration information click here.