Empirical Legal Studies

Assessing Primary and Secondary Effects of Sobriety Checkpoints

While police use of sobriety checkpoints has become increasingly common since the 1980s, legal discomfort, particularly Fourth Amendment concerns, persist. While the SCOTUS concluded in its Sitz decision (1990) that sobriety checkpoints' social benefits outweighed costs on individual liberties, states remain free to preclude their use. Indeed, 11 states have ruled sobriety checkpoints illegal.

Aside from their legal status, empirical questions about sobriety checkpoints' efficacy lurk. In an intriguing new contribution to the emerging empirical literature, Sobriety Checkpoint Laws, Fatal Cars Crashes and Arrests, Lauren Jones (OSU--Public Policy) and C. Blain Morin (OSU--Public Policy) explore the effects of a state removing sobriety checkpoints on fatal car crashes and arrests.

Leveraging data from the Fatality Analysis and Reporting System and Uniform Crime Reports, the authors execute a twoway fixed effects (TWFE), event study analysis. As the paper notes, the twoway fixed effects analysis extends a difference-in-difference specification, where changes in outcomes across pre-treatment and post-treatment time periods are compared, across a treated unit and a control unit. As the abstract (below) makes clear, the paper's key findings include an annual 17 percent increase in fatal crashes and a 25 percent increase in the number of DUI arrests in states that cease using sobriety checkpoints.

“Sobriety checkpoints have been used since the early 1980s to enforce laws prohibiting driving while intoxicated (DWI), and to deter DWI. However, their constitutionality has been questioned, as they allow police to stop drivers without cause. As a result, eleven states have banned their use. Using a two-way fixed effects event study analysis, we estimate the effect of rendering sobriety checkpoints illegal on state-level counts of fatal car crashes, and arrests for driving under the influence of drugs or alcohol (DUI), and drug possession. We find evidence that when states make sobriety checkpoints illegal, the number of annual fatal crashes increases by 17 percent. We also find evidence that rendering sobriety checkpoints illegal leads to a 25 percent increase in the number of annual DUI arrests in a state, suggesting that the deterrent effect of checkpoints is much stronger than their enforcement impacts. We also find suggestive evidence that eliminating sobriety checkpoints increases racial disparities in DUI and cannabis possession arrests, with larger impacts on arrests among Black residents.”

Law School Curriculum: Reform, Assessment, and Research Design

While age-old complaints persist about the basic structure of the first-year law school curriculum, few law schools have dared to experiment with meaningful reforms. Fewer still study these reforms coherently. The "Curriculum B" experiment at Georgetown Law School, as well as a recent paper by David Hyman et al. (Georgetown), Does the 1L Curriculum Make a Difference?, are important and interesting exceptions.

Since 1991, Georgetown Law School has offered an "alternative" Curriculum B along side a standard, more traditional first-year curriculum. Curriculum B was designed to provide “critical” perspectives on the law and respond to important changes in legal practice and theory in the years since the standard 1L curriculum was first developed decades earlier. At Georgetown, students elect into Curriculum B and, when demand exceeds supply, student assignment to Curriculum B is determined by lottery.

For assessment purposes, the quasi-randomized control trial aspect of student assignment to Curriculum B provides a way to overcome selection bias. That is, Hyman et al. compare student outcomes for those who opted into Curriculum B (the treatment group) with student outcomes from those students who wanted to receive the "treatment" (Curriculum B) but, due to excess demand and random selection (lottery), were instead assigned the "placebo" (that is, to the standard, traditional first-year curriculum).

Aside from careful attention to research design, a few of the paper's warrant note. When it comes to students' overall law school academic performances, for example, the authors find that "Curriculum B leads to a mean preserving contraction in students’ overall grades. At the same time, we find that it leads to diminished performance in two key business law electives (Corporations and Securities Regulation)."

Business Litigants and the Roberts Court

Assessing “How pro business is the Roberts Court?” is both straightforward and complicated. Based on how business litigants fare in the Roberts Court versus non-business litigants over time, business does quite well. The win rate for business in the Roberts Court, 63.4 percent, is 15 percentage points higher than the next highest rate of business wins over the past century (the Rehnquist Court, at 48.3 percent). While it may be tempting for many to conclude that the current pro business tilt is purely a function of a Republican majority of justices on the Roberts Court, as Lee Epstein (USC) and Mitu Gulati (Virginia) argue in a recent paper, A Century of Business in the Supreme Court, 1920-2020, “this is only the beginning of the story.”

Epstein and Gulati bring data from the Supreme Court Database to the question. Starting with the 11,121 orally argued cases resulting in a signed opinion or judgment between the 1920-2020 terms, their paper focuses only on those 4,070 cases (35,173 votes) in which business was the petitioner (appellant) or respondent (appellee). That is, they excluded cases in which business was on both sides of a legal dispute.

Their core findings defy conventional wisdom and inject important complicating wrinkles. What they find includes: "(a) It is not just the Republicans on the Roberts Court who are more pro business than in prior Courts, but the Democrats as well; (b) The Government, through the SG’s office and across both Democratic and Republican administrations, has been much more supportive of business positions than in prior eras; and (c) An elite Supreme Court bar has emerged in recent years and businesses have hired them disproportionately so as to better influence the Court."

Empirical Contours of the Political Question Doctrine

Amid the enduring and varied debates about the political question doctrine, motivated by the Supreme Court's 1962 Baker v. Carr decision, the scholarly commentary shares one key attribute. Specifically, it dwells on the Supreme Court. Such a focus is ironic as while the Court has only rarely applied the political question doctrine, lower federal courts, by contrast, have invoked the doctrine "as a basis for dismissal in hundreds of cases." Drawing on both qualitative and quantitative (descriptive) methods, in a recent paper, The Real Political Question Doctrine, Curtis Bradley (Chicago) and Eric Posner (Chicago) explore how the political question doctrine has operated in the lower courts since the Supreme Court's Baker decision.

The authors find that lower courts use the doctrine to "evaluate their own institutional capacity to resolve politically sensitive disputes. It is the lower courts’ more limited capacity compared to the Supreme Court that explains the lower courts’ heavier reliance on the doctrine." More specifically, "the political question doctrine is a screening mechanism that the lower courts use to take account of limits on their institutional capacity. The Supreme Court has less need for this mechanism, so most of the doctrine’s life must be found in the foothills rather than at the summit of Mount Olympus. From this standpoint, the doctrine is not moribund but full of life. It is not limited in the ways suggested by the 'classical' or 'functional' theories. The lower court practice is best captured by Bickel’s prudential theory." An excerpted abstract follows.

"... We provide the first empirical account of how the doctrine has operated in the lower courts since Baker. Our account is based on both a quantitative analysis of a sample of these decisions and on a qualitative review of these and other decisions. This account reveals a political question doctrine that is substantially different from the one described in most scholarship: it is more vibrant, heavily focused on foreign affairs, often applied in non-constitutional cases, more prudential, and not a permanent disallowance of judicial review...."

The Impact of Increased "Sunlight" on Police Traffic Stops

Complicated legal and policy issues incident to policing--and conflicts with citizens the police serve--persist and continue to motivate empirical work. One under-studied aspect, however, involves the degree to which, if any, adverse (digital) media coverage of instances of police brutality informs police conduct going forward. Policing the Police: Examining the Role of News
Reports on Racially-Biased Policing reflects a recent effort at this difficult research question.

In their paper, Uttara Ananthakrishnan (Wash.--Business) et al., set out to assess the impact of public media accounts of police misconduct/violence from January 2013 to May 2018 (N=6,026) on police-initiated traffic stop data drawn from 227 counties across 18 states. The authors execute a DiD model that, because police traffic stop outcomes are "naturally tied to counties," compares "the change in the number of police stops in counties exposed to news reports with the same change in control counties that did not have news reports about their policing incidents during the same period.” Among the paper's findings is that news reports on fatal uses of police force on minorities have a "negative impact on police stops of minorities but not on Whites, on average.” An excerpted abstract follows.

"... However, it is theoretically unclear whether the reporting of excessive use of police force on minorities can have a tangible impact on subsequent policing outcomes. In this paper, we aim to answer the question of whether and how digital news on police brutality is effective in shaping subsequent police actions. To address this question, we construct a cross-sectional dataset of news reports of police violence and police traffic stop records. Under a difference-indifference framework, we find that news reports on police brutality reduces police stops of minorities. Additionally, we find that news with sad frames are more effective in effectuating change in policing behaviors. Finally, we learn that the impact of news reports are less effective in minority-dominated areas and high-crime areas."

Data on Naturalization

Naturalization grants immigrants a host of new and important rights, privileges, and opportunities. While a growing empirical literature focuses on such issues as who is willing to naturalize and why, barriers to seeking naturalization, and the impact of obtaining citizenship on the social, economic, and political integration of immigrants, comparatively less is known about government determinations of who is approved or denied once a naturalization application is submitted to the U.S. Citizenship and Immigration Services (USCIS).

Drawing on new administrative data obtained from USCIS (N=2,687,101 non-military applications adjudicated between October 2014 and March 2018), in their paper, The Importance of Race, Gender, and Religion in Naturalization Adjudication in the United States, Emily Ryo (USC) and Reed Humphrey (Wash.—pub. policy) find significant group disparities in naturalization approvals based on applicants’ race/ethnicity, gender, and religion, controlling for individual applicant characteristics, adjudication years, and variation between field offices.

More specifically, the paper finds that “non-White applicants and Hispanic applicants are less likely to be approved than non-Hispanic White applicants, male applicants are less likely to be approved than female applicants, and applicants from Muslim-majority countries are less likely to be approved than applicants from other countries. In addition, race/ethnicity, gender, and religion interact to produce a certain group hierarchy in naturalization approvals. For example, the probability of approval for Black males is 5 percentage points smaller than that of White females. The probability of approval for Blacks from Muslim-majority countries is 9 percentage points smaller than that of Whites from other countries. The probability of approval for females from Muslim majority countries is 6 percentage points smaller than that of females from other countries. This study contributes to our understanding of the nature of inequalities present in agency decisionmaking in the naturalization process.”

Everything You Wanted to Know About Tables (in Stata)

Throughout the second-half of 2021, the Stata Blog ran a 7-part series on generating various tables. A single link to the 7-part series is here.

Multi-Level Model "vs." Clustering

As this question (or something like it) is not an infrequent one, I found Jeff Wooldridge's response (No.2) in this StataList discussion (here) quite helpful. The bottom line is that decisions incident to estimating a multi-level model or/and clustering pivot on an array of considerations.

CELS -- UVA -- 2022: Papers -- Deadline Extended

EXTENDED Paper Submission Deadline: Sunday, July 10, 2022 (11:59 PM Eastern Daylight Time).

To submit a paper, please go to the CELS UVA 2022 website (click here; submissions tab, click here). There is no charge for submissions.

Hotels: It is likely to be a crowded weekend in Charlottesville, so please be sure to reserve your hotel early by following the Hotels tab on the conference website.

About: The University of Virginia School of Law and the Society for Empirical Legal Studies (SELS) are pleased to host the 16th Annual Conference on Empirical Legal Studies (CELS) on November 4-5, 2022, in person at the University of Virginia in Charlottesville. CELS is a highly regarded interdisciplinary gathering that draws researchers from across the United States and the world, bringing together scholars in law, economics, political science, psychology, policy analysis, and other fields who are interested in the empirical analysis of law and legal institutions. Papers are selected through a peer review process, and discussion at the conference includes assigned commentators and audience questions.

Scope: CELS will consider empirical papers spanning all areas of empirical legal studies. Authors are encouraged to submit works-in-progress; however, submissions should be completed drafts that include principal results. Submitted papers must be unpublished (and expected to be unpublished at the time of the conference). If accepted, authors will have an opportunity to submit a revised draft prior to the conference for presentation and discussion. Please note that accepted papers will be made available to all conference participants.

Keynote: We are thrilled to announce that the keynote Friday dinner speaker will be Dr. Anne-Marie Slaughter, CEO of New America, Bert G. Kerstetter '66 University Professor Emerita of Politics and International Affairs at Princeton University, and former Dean of Princeton University's School of Public and International Affairs. There is a separate charge for attendance to this dinner event.

If you have questions about the submissions process, please contact the conference organizers at cels2022@law.virginia.edu.

Simple Problems, Simple Solutions

While I should have known this years ago, better late than never.

As one becomes familiar with the broad descriptive contours of new data, the tab function is both well-known and ubiquitous. Finding analogous basic descriptive statistics for sub-samples, while certainly possible, was typically (for me, anyway) a bit cumbersome with the tab command.

A recent discussion on the StataList, however, introduced me to the simple (and code-saving) power of a related command, tabstat. As this discussion illustrates (click here), what the tab command accomplishes with multiple lines of coding and multiple output tables, the tabstat command, by contrast, accomplishes with a single line of code and a single, unified output table.

A "Guide" on how to Interpret Empirical Corporate Governance Scholarship

While empirical work continues to proliferate across an growing array of legal "silos," lawyers (and some law professors) increasingly need to engage with these studies to understand their implications for legal practice and policy implications. This is especially true for corporate governance scholars (and practitioners). As empirical corporate governance work reaches a wider audience, the work intersects with an ever-increasing number of individuals "not trained in econometrics and statistical methods."

This is a problem, and a recent paper endeavors to contribute to a solution. In A Lawyer's Guide to Empirical Corporate Governance, Ofer Eldar (Duke) sets out to provide a general guide that overviews "key empirical strategies for evaluating the relationship between governance and shareholder value, and to demonstrate to lawyers how they can engage with their weaknesses." Critically, the paper identifies a set of three key comparisons that dominate empirical corporate governance scholarship: "(1) comparing firms with and without the governance provisions, (2) evaluating the value of firms before and after adoptions or removals of governance provisions, and (3) evaluating what happens to shareholder value after a legal change."

While papers like these may not be standard empirical legal scholarship, they nonetheless contribute to the field more generally by helping to make empirical scholarship a bit more accessible to those lacking formal empirical training. The abstract follows.

"Empirical studies in corporate law have proliferated in the last two decades, and have had a major impact on legal practice and policy relating to corporate governance. As a result, lawyers increasingly engage with these studies in order to understand their implications for legal practice and policy. This is often challenging because most lawyers are not trained in econometrics and statistical methods, and thus, there is a risk that they misinterpret these studies and their implications. These studies, while critical for evaluating different corporate governance regimes, suffer from well-known weaknesses. The purposes of this guide are to provide an overview of the key empirical strategies for evaluating the relationship between governance and shareholder value, and to demonstrate to lawyers how they can engage with their weaknesses. The main theme of the guide is that largely all empirical strategies are based on three key intuitive comparisons: (1) comparing firms with and without the governance provisions, (2) evaluating the value of firms before and after adoptions or removals of governance provisions, and (3) evaluating what happens to shareholder value after a legal change. By identifying these basic comparisons and the roles they play in the key methodologies for evaluating shareholder value, lawyers will be better positioned to assess empirical studies and evaluate their implications for legal policy."

Law Firms, M&A Work, Gender, and the Uneven Credentials Burden

It remains well-understood that while women have made up approximately one-half of law school classes over the past decades as well as the ranks of entry-level law firm associates, attrition is uneven across genders throughout the path to law firm partnership. No doubt this uneven attrition is the product of many complex variables and factors, some more troubling than others. What is less well-understood, however, is how attrition distributes post-law firm partnership.

In a recent paper, Gender, Credentials and M&A, Tracey George (Vanderbilt), Albert Yoon (Toronto), and Mitu Gulati (Virginia) explore this topic with data from one specific practice area: M&A. The specific practice area (M&A) is important as "[r]elative to a number of other practice areas, such as the issuance of corporate or municipal bonds or managing trusts and estates, M&A tends to be a “feast or famine” area. Lawyers are either busy working around the clock (if they are trying to close a deal) or are twiddling their thumbs.” And this aspect is salient for gender comparisons as "a high degree of variability in the demands from work tends to hurt more those workers who have bigger fixed outside obligations that cannot easily be moved or postponed (e.g., caring for small children or elderly parents).”

The paper executes a two-stage empirical strategy. It first explores the "universe" of M&A attorneys. The second stage analyzes who becomes a "lead deal lawyer" (representing either the buyer or the seller). One key finding is that while at both stages lawyer credentials matter, they "matter more" for women (emphasis added). The abstract follows.

“For the past several decades, women have made up roughly half of law school classes and the ranks of entering law firm associates. Attrition between entry to law firms and partnership results in women comprising 20 and 25 percent of partners. But is there yet more attrition to the top of the partnership pyramid? Analyzing the past decade of data on publicly-filed M&A deals and detailed biographical information of M&A lawyers, we find that women make up fewer than 10 percent of deal leaders. When we look at the factors that determine who becomes a deal leader, we find that credentials – both educational and professional – matter. But they matter more for women. And one credential – attending a top law school – seems to matter a lot. Using conversations with senior lawyers, we try to get answers for why.”

Comparing Regression Output Across Models

As I am seeing more papers trying to do so (as well as discussions about strategies on how to do so), I thought links to a recent StataList discussion as well as some web-based resources might be of general interest. The basic set-up follows:

Model A: Y = α1 + β1X1 + Σθ∙C + firm-fixed effect + year-fixed effect.

Model B: Y = α1 + β2X2 + Σθ∙C + firm-fixed effect + year-fixed effect.

Σθ∙C is list of control variables and are the same for both model A & B.

Question: How to compare the coefficients of β1 and β2?

One approach exploits a post-estimation command in Stata (suest), and then compares means with test (for a discussion and illustration, click here). Another illustrated example, courtesy of the UCLA Statistical Methods and Data Analytics site, is here.

Moving From "Data to Policy Recommendations"

For many scholars and policymakers, “short-termism” behavior in the stock markets is assumed to be a problem. Unsurprisingly, researchers have empirically assessed this issue to better understand its dynamics, extent, and contours.

In a recent paper, Stock Market Short-Termism: What the Empirical Evidence Tells Policymakers, Mark Roe (Harvard) does not contribute more results. Instead, Roe takes a step back and presents a helpful example for empiricists in terms of how to interpret and contextualize results from multiple studies looking at the same general issue (here, stock market short-termism). That is to say, this paper’s contribution speaks to the discrete task of moving from “data to policy recommendations.”

Specifically, Roe considers “what the evidence directly tells us about stock-market-induced short-termism and how that evidence must be evaluated conceptually to reach an economy-wide assessment for policymakers.” What he finds is that for “stock market short-termism, it is harder to move from the data to policy recommendations than is usually thought.” Roe notes an array of factors contributing to the degree of difficulty, including evenly divided academic evidence and varied (and limited) research designs that invite external validity questions.

While empirical legal scholars appropriately focus on executing a coherent research design and generating stable, robust results, additional moves remain. And one of these, for some empirical legal scholars, involves a move from the “data to policy recommendations.” What Roe’s paper helpfully emphasizes is that these secondary and tertiary moves introduce an additional set of challenges and complications that warrant greater attention. An excerpted abstract follows.

“… The typical research effort seeks to measure whether a local treatment induces more local short-termism, not whether the economy-wide impact is severe. But evaluating short-termism’s economy-wide impact is essential for policymaking; policymakers must consider whether the economy is failing to invest or cutting back on R&D because of a stock market afflicted with a truncated time horizon. Local findings (of the impact on a subset of firms with a particular characteristic, such as short-vesting stock options, rapid stock market trading, or hedge fund activism) need not scale to economy-wide results; some local results will do so only serendipitously; and, finally, there are structural reasons why for stock market short-termism one should expect economy-wide results not to match local results. More than most corporate law issues, the short-termism problem faces a high failure-to scale hurdle.”

Data on K-12 School Suspension Reform Efforts

Incident to the federal Gun Free School Act (1994), all fifty states adopted an array of "zero-tolerance" policies, which include out-of-school suspensions ("OSS"), for various forms of student misconduct. As evidence of adverse consequences mounted, some states, including Rhode Island, limited school districts' use of out-of-school suspensions. In 2012, Rhode Island began prohibiting out-of-school suspensions for "low-level" student infractions and, in 2016, began requiring school district-level reviews of whether out-of-school suspensions raise any long-standing EPC concerns.

In a recent paper, Do School Suspension Reforms Work? Evidence from Rhode Island, Terry-Ann Craigie (Smith College--econ) "exploits student-level administrative data from the Rhode Island Department of Education (AY 2009–2010 to AY 2017–2018), along with a triple-difference (DDD) framework, to measure the impact of each suspension reform on OSS [out-of-school suspension] outcomes for treatment infractions in treatment schools (i.e., schools and within-school grades that used OSS to penalize treatment infractions, ex ante). The study also employs a quadruple-difference (DDDD) model to measure the extent to which racial-ethnic disparities in OSS changed for the treatment groups under each reform."

Results from the study imply that Rhode Island's initial (2012) reform "substantially lowers OSS for attendance-specific infractions in treatment schools." Results on Rhode Island's 2016 reform designed to address distributional concerns, however, are "mixed." The paper's abstract follows.

“In Rhode Island, out-of-school suspensions were excessively and disproportionately used to penalize low-level infractions. To address this problem, the Rhode Island General Assembly passed legislation, effective May 2012, prohibiting out-of-school suspensions for attendance-specific infractions. Four years later, they passed additional legislation to curb out-of-school suspensions for disruption-specific infractions. This study examines the impact of these suspension reforms on out-of-school suspension outcomes for treatment infractions and corresponding racial-ethnic disparities. To execute the analyses, the study uses student-level administrative data (AY 2009–2010 to AY 2017–2018) from the Rhode Island Department of Education, along with quasi-experimental estimation. The study finds that only the first reform lowers out-of-school suspension outcomes for attendance-specific infractions and corresponding racial-ethnic disparities.”