Empirical Legal Studies

The Two-Part Metascience 2023 Confernece(s)

At the request of a loyal Elsblog reader--who is also helping organize the Metascience 2023 conference--I pass along the following.

Metascience is an organization that promotes "global gathering for knowledge sharing, community building, and opportunities to define a roadmap of research and intervention priorities to accelerate science." To this end, "Metascience 2023" involves two separate--though related--conference components. The in-person component is scheduled for 9-10 May 2023, at the National Academy of Sciences in Washington, DC (USA). The organizing group is also mounting a free virtual component, set for "late April and early May," 2023.

For anyone interested in additional information, click here for general conference information, and here for the conferences' call for papers/proposals.

Civilian Complaints Against Police & Prosecutor "Upcharging"

Citizens’ engagement with police serves as a valuable resource. One particular form of citizen engagement—civil complaints—can impose a serious cost to the targeted police officer as well as the police department. To try to reduce the potential negative effects of a citizen complaint against law enforcement agents, other criminal justice actors could act to re-frame the encounter such that the optics look unfavorably on the complaining citizen. Since a defendant cannot seek monetary damages in civil court unless his/her conviction or sentence was declared invalid, prosecutors may increase the number or the severity of charges that the defendant faces to propel the defendant toward accepting a plea deal, which would prevent the defendant from seeking further financial reparations. In a new paper, Do Civilian Complaints Against Police Get Punished?, DeAngelo (Claremont Colleges) et al. pursue this question empirically.

To do so, the authors examine arrest, complaint and charging data from Cook County (Illinois). The results indicate that a positive, causal link between citizen complaints and an increased number of total charges filed against a defendant exists. The paper also notes that a civilian complaint yields a larger plea discount, likely to induce a plea agreement. Notably, the main findings are entirely driven by charges filed directly by the prosecutor’s office rather than directly by the police. Given that prosecutors are likely filing charges after civilian complaints have been filed against law enforcement, it appears that prosecutor behavior is leading to upcharging in response to civilian complaints and not police decisions. An excerpted abstract follows.

“… This paper empirically investigates one way in which criminal justice institutions respond to civilian complaints. Namely, criminal prosecutors can upcharge a defendant who files a civil complaint against law enforcement. By upcharging, the prosecutor can increase the likelihood that a defendant will accept a plea deal, thus preventing the defendant from seeking monetary damages in civil court (Heck vs. Humphrey, 1994). Using data on citizen complaints and criminal charge outcomes from Cook County (Illinois), we find a strong causal link between a citizen filing a complaint and the total number of charges filed.”

The Empirics of Sobriety Checkpoint Stops

Motor vehicle crashes remain the leading cause of death for Americans under age 25, and many of these crashes involve substance use. To help reduce the volume of impaired drivers, many--but not all--states allow for sobriety checkpoints (SC). While historically checkpoint stops were used to enforce drunk driving laws, they are increasingly being used to deter and prosecute drugged drivers, especially in states where marijuana is legal.

Although “no-cause” traffic stops remain controversial, comparatively little data exist that helps inform related debates. While the SCOTUS permits such stops as a general matter, courts in eleven states have ruled that state constitutional provisions preclude sobriety checkpoints. And those states that discontinued sobriety checkpoints provide a critical source of data that a recent paper, Sobriety Checkpoint Law, Fatal Car Crashes and Arrests, exploits. In their paper, Lauren Jones (OSU—public affairs) and C. Blain Moran (OSU—public affairs) compare trends in event frequencies “across states that did and did not turn off their SC laws, net of confounding factors.” The paper finds, at bottom, that when states make sobriety checkpoints illegal, annual fatal crashes and DUI arrests increase. The paper also notes a racial effect relating to DUI and cannabis arrests. The abstract follows.

“Sobriety checkpoints have been used since the early 1980s to enforce laws prohibiting driving while intoxicated (DWI), and to deter DWI. However, their constitutionality has been questioned, as they allow police to stop drivers without cause. As a result, eleven states have banned their use. Using a two-way fixed effects event study analysis, we estimate the effect of rendering sobriety checkpoints illegal on state-level counts of fatal car crashes, and arrests for driving under the influence of drugs or alcohol (DUI), and drug possession. We find evidence that when states make sobriety checkpoints illegal, the number of annual fatal crashes increases by 17 percent. We also find evidence that rendering sobriety checkpoints illegal leads to a 25 percent increase in the number of annual DUI arrests in a state, suggesting that the deterrent effect of checkpoints is much stronger than their enforcement impacts. We also find suggestive evidence that eliminating sobriety checkpoints increases racial disparities in DUI and cannabis possession arrests, with larger impacts on arrests among Black residents.”

Theory's Role in ELS

A not infrequent, though diminishing, canard is that "too many" empirical legal studies papers are "atheoretical." While the claim itself is an empirical one (and one that I side-step), a larger, broader point endures. Specifically, the best of ELS is certainly informed by well-developed theory. In a recent paper, The Role of Theory in Empirical Legal Studies, Tonja Jacobi (Emory) makes this case in a brief and widely accessible paper. While perhaps targeted to "early career" ELS scholars and non-technical consumers of empirical research, the paper helpfully reminds all of theory's essential role in empirical legal studies. The paper's abstract follows.

"The growing field of empirical legal studies is not an alternative to theory: within every empirical inquiry is an implicit formal theoretical model. Further, good empirical tests should be inspired by substantive theories of the phenomena being analyzed. This essay explains how theory is built into all empirical testing and illustrates why empirical tests are best informed by well-developed theories. It uses examples of empirical studies of both US Supreme Court and Australian High Court oral argument to illustrate how these issues apply to the central issues at the intersection of empirical legal studies and constitutional law: the role of judicial ideology and strategy, and predicting judicial behavior. The essay explores common errors in both conducting and consuming empirical work, and presents some solutions, including both methodological fixes and how to draw on substantive expertise. It aims to help the non-technical reader understand when counterintuitive results should be queried and when, in contrast, resistance is more reflective of priors of the reader rather than potential weaknesses of the work. And it acts as a reminder to technical readers and empirical practitioners that theory is built into every empirical test, and so must be taken seriously."

CELS 2023--Univ. of Chicago Law School

Coming on the heels of an exceptionally well-executed and successful CELS at Virginia, CELS 2023 co-organizers--Adam Chilton, Dhammika Dharmapala, and William Hubbard--announced that CELS 2023 is scheduled for 13-14 October 2023, at the University of Chicago Law School. More information will follow.

Case Outcomes & Judge Financial Conflicts

To assist with insuring that "justice is blind," a presiding judge is expected to recuse from any case where he/she is financially conflicted with any of the litigating parties. Evidence that this expectation is not being consistently met was widely reported in 2020-2021. Indeed, the public controversy elicited a response from Chief Justice John Roberts. In his 2021 annual report on the federal judiciary, and in direct response to the controversy surrounding numerous federal judges, Chief Justice Roberts downplayed the controversy’s significance, characterizing the incidents as "isolated violations" and "unintentional oversights."

Notwithstanding Justice Roberts' characterizations, given the widespread implications posed by judges' failures to recuse when proper it is important to understand whether these recent episodes are one-time incidents or point to a larger, systemic problem. To address this question, in a recent paper, Judge Financial Holdings and Case Outcomes: Evidence from Judge Financial Disclosures, Tuhin Harit (Univ. Dallas--Mgmt.) et al. examine whether there is a relation between judges' disclosed financial holdings and the outcomes of the trials over which they presided.

Drawing from a sample that includes cases from 2000–2021, the paper finds that “stock ownership by judges significantly improves the chances of a litigant's winning a case.” In cases where the presiding judge is financially conflicted with either the defendant or plaintiff, “their chance of a favorable decision improves by 3.3% and 4.3%, respectively, after controlling for judge, industry, district, case file year, and firm characteristics.” An excerpted abstract follows.

"... Using novel data on financial disclosures of judges and the civil case information of public firms across district courts in the United States between 2000 and 2021, we document multiple instances in which judges fail to recuse themselves in cases where they hold common stock. We find robust evidence that judges rule more favorably and take longer when deciding over conflicted cases as compared to unconflicted ones. We also find that investors are positively surprised by these outcomes, as reflected by stock returns to trial outcomes. Collectively, our results provide the first large-scale evidence of the relation between judges' financial holdings and case outcomes.”

Dodd-Frank, Accounting Fraud, and Whistleblowing

Owing to several notable financial frauds involving whistleblowers, regulators took steps to strengthen whistleblowing provisions. One prominent example is the Dodd-Frank whistleblower program which provides financial rewards to whistleblowers who report financial fraud to the SEC. While regulators may believe that the Dodd-Frank whistleblower law helps deter accounting fraud, whistleblower tips can be frivolous and processessing such tips is not cost-less. Similar to all too many public policies, little empirically is known about the the Dodd-Frank whistleblower provision's efficacy.

In a recent paper, Did the Dodd-Frank Whistleblower Provision Deter Accounting Fraud?, Philip Berger (Chicago--Booth Business) & Heemin Lee (Baruch College) examine whether the Dodd-Frank whistleblower law has a causal effect on the deterrence of fraud and quantify the size of the deterrence effect. Exploiting a control group of firms previously exposed to state-level false claims statutes, the paper sets out to assess the "underlying (or ex ante) likelihood of accounting fraud decreases for firms subject to Dodd-Frank and not previously exposed to the risk of whistleblowing" via analogous state laws. At least as it relates to Dodd-Frank's whistleblowing component, the paper a finds a decrease in the probability of fraud attributable to whistleblower provisions. The abstract follows.

“We examine the deterrence effect of the Dodd-Frank whistleblower provision on accounting fraud. To facilitate causal inference, we use state False Claims Acts (FCAs), under which whistleblowing about accounting fraud at a firm invested in by a state's pension fund can result in monetary rewards from that state’s government. We divide our sample into firms exposed and not exposed to whistleblowing risk from a state FCA during the 2008 – 2010 period that preceded the 2011 SEC implementation of the Dodd-Frank whistleblowing provision. We hypothesize that firms already exposed to a state FCA whistleblower law are less affected by the Dodd-Frank whistleblower provision. Using the companies exposed to a state FCA as control firms in our Dodd-Frank tests, the remaining firms constitute the treatment sample. We find that exposure to Dodd-Frank reduces the likelihood of accounting fraud of treatment firms by 12% to 22% relative to control firms, but do not find that it affects audit fees.”

Med Mal Insurance Premia & Economic Theory

The medical malpractice insurance market is inherently complex and made even more so by the growing number of state efforts to regulate it. Complexities aside, however, standard economic theory should, in the main, explain, to some degree, med mal premia trends. Findings from a recent paper, How Do Insurers Price Medical Malpractice Insurance?, by Bernie Black et al. (Northwestern), however, "are not easily explained by standard economic theory."

Prof. Black et al.'s paper analyzes almost three decades worth of med mal premia data (gathered by the Medical Liability Monitor). Core findings include:

• Med mal premia are only loosely connected to insurer costs, even over long periods;
• Premia are much higher in states with damage caps, even after controlling for costs; and
• The Premium/Cost Ratio varies widely both across states at a given time and within states across time.

Taken together, the paper's findings imply that med mal insurance companies do not fully adjust premia to changes in direct costs even over long time periods as standard economic theory might predict.

Poisson Tutorial Videos

The good folks at Stata recently posted a series of video tutorials focusing on how to fit various Poisson regression models which assist when modeling count data. The various videos take up a Poisson model with: a) continuous predictors (here); b) categorical predictors (here); and c) Poisson models that include both continuous and categorical variables (here).

4th annual Association for Interdisciplinary Metaresearch and Open Science Conference (AIMOS2022) -- Registration

At the request of an ELS colleague, I want to help circulate the following registration information for the 4th annual Association for Interdisciplinary Metaresearch and Open Science Conference (AIMOS2022). Please note that the conference is scheduled for November 28-30, 2022, at the University of Melbourne. AIMOS2022 will be an in-person conference, with support to attend virtually.

As an association, AIMOS aims to make the research process more trustworthy and efficient and to promote the study of how research is done and how it can be improved.

The conference program will also explore other areas of metaresearch including a host of mini-note sessions on:

• Big Team Science
• Credible Science for Decision Making
• Correcting the Scientific Record
• History of Metascience 
• Unpacking Trust in Science
• The Many-Analysts EcoEvo Project
• The Systematizing Confidence in Open Research and Evidence (SCORE) Program.

In addition to invited speaker sessions, submissions are now open for: discussions about norms, practices, and cultures with science and scholarship more broadly; learning and development through practical skills workshops in open science (e.g., R, lab notebooks, pre-registration); and getting things done in hackathons.

Finally, a small number of AU$1,000 travel grants are also available to support in-person attendance.

A Small--But Important--Logistical Matter Incident to Dataset Merging

Those who have merged (or appended or crossed) datasets may have inadvertently bumped into a not uncommon problem--where the two separate datasets share a variable with the identical variable label name. For relatively smaller datasets, and where the researcher is aware of this hurdle ex ante, one can always, by hand, rename variables prior to combining the datasets. For larger datasets with many more variables, however, while this may not be practical it nevertheless remains critical.

A recent discussion of how to manage this nettlesome problem is found here. The comments' leading suggestion references Daniel Klein's -elabel- package, available from SSC (as it involves user-written code). Put briefly, the elable command allows a user to add a suffix (e.g., "_1") to every variable name in "dataset1." An analogous manipulation to datast2 can add another unique suffix (e.g., "_2") to every variable name in dataset2. And so forth. The end result includes datasets with distinct variable label names that can then be combined.

Helpful Statistics Resources

On the better late than never front, I just stumbled across a wonderfully rich "one-stop" resource for statistics novices. Germán Rodríguez (Princeton) hosts a site (here) that includes tutorials for both Stata and R. As to the tutorials' scope, the Stata tutorial, e.g., ranges from a "gentle introduction" to coding examples for customizable tables. Many--especially students--may find this resource helpful.

Firearms' Preferences and an Empirical Analysis of Secondary and Tertiary Regulatory Consequences

Civilian (household) gun ownership levels continue rise along with efforts to regulate gun ownership and possession. Scholarly efforts in this space, while increasing, confront complex data, modeling, and research design challenges.

In Preferences for Firearms and Their Implications for Regulation, the authors, Sarah Moshary (UC Berkeley—Business) et al., steer clear of trying to identify causal links between gun ownership and crime or gun deaths. Instead, the paper "estimates the effects of policy on both the number and types of guns sold in the primary market, as well as the consumer surplus that accrues to gun owners from their purchases.”

Among the various policy counterfactual simulations that the paper explores include a policy that bans assault weapons. What the paper's findings imply is a consumer shift from assault weapon ownership to firearm ownership. This is so, the paper argues, because even though consumers who are in the market for handguns do not consider long guns at all, many consumers that consider purchasing a long gun, by contrast, "are also interested in buying a handgun." As the "vast majority" of gun violence involves handguns, however, this substitution effect, fueled by a regulatory ban, introduces complicating policy wrinkles. The paper's abstract follows.

“This paper estimates consumer demand for firearms with the aim of evaluating the likely impacts of firearm regulations. We first conduct a stated-choice-based conjoint analysis and estimate an individual-level demand model for firearms. We validate our estimates using aggregate moments from observational data. Next, we use our estimates to simulate changes in the number and types of guns in circulation under alternative regulations. Importantly, we find that bans or restrictions that specifically target ‘assault weapons’ increase demand for handguns, which are associated with the vast majority of firearm-related violence. We provide distributions of consumer surplus under counterfactuals and discuss how those distributions could be useful for crafting policy."

Checks on Prosecutorial Power

In light of the significant authority that prosecutors wield, we want them to be, at once, both accountable and independent. Obviously, these twin goals are structurally at odds with each other. "Electing prosecutors ensures that they are accountable, but it limits their independence. In contrast, appointing prosecutors increases independence, but it does so at the expense of accountability." In the U.S., 45/50 states elect local prosecutors "who bear the primary responsibility for bringing felony prosecutions in their jurisdictions." However, as Carissa Hessick (UNC) et al. note in a recent paper, Understanding Uncontested Prosecutor Elections: "[prosecutor] elections in the United States generally fail to capture the accountability benefits of elections because so many prosecutor elections are largely uncontested and uncompetitive."

Drawing partly on data gathered by the Prosecutors and Politics Project, the paper models "contested" prosecutor elections. In a slight break from prior findings from related literatures, this paper's core findings emphasize the salience of whether an incumbent is running and the jurisdiction's population. The paper's (excerpted) abstract follows.

"... Using a logistic regression analysis, this Article estimates the individual effects of a number of variables on prosecutor elections. It finds that several factors that have been previously identified as contributing to uncontested election are not, in fact, what drives uncontested elections for prosecutors. Instead, the factors with the largest effect are whether an incumbent runs and the population of the district. It also identifies two features of state election law that contribute to the dearth of contested elections. The Article concludes by noting that these factors suggest specific policy changes that could help to increase the number of contested and competitive elections—thus ensuring that voters can help guide important criminal justice decisions in their communities."

The Empirics of Choice of Law

In an effort to peer into judicial decisionmaking more generally, a recent paper by Dan Klerman (USC), Bias in Choice of Law: New Empirical and Experimental Evidence, levers both observational and experimental data drawn from the vehicle accident context to examine choice of law disputes. The paper considers three leading sources of choice of law bias: pro-resident, pro-plaintiff, and forum law. The findings intrigue and complicate prevailing conventional wisdoms.

Specifically, core findings include that while judges seem to favor the plaintiff, judge decisions do not tilt in favor of application of the law they are most familiar with (the law of the forum). While state court judges favor local residents, federal court judges do not. Although the traditional “place of the injury” choice-of-law rule was more constraining than modern American choice-of-law standards (such as the Restatement Second), even judges under the traditional rule favored the plaintiff more often than seemed warranted. The abstract follows.

"Scholars have argued that modern American choice-of-law is subject to three biases: (1) a bias in favor of residents of the forum state (pro-resident bias), (2) a bias in favor of plaintiffs (pro-plaintiff bias), and (3) a bias in favor of the law of the forum (forum law bias). This article brings new evidence about these biases from a comprehensive database of vehicular accident cases and from the first experiment related to choice of law. Overall, there is some evidence of a pro-resident bias in state (but not federal) court, some evidence of pro-plaintiff bias in both federal and state court, and no evidence of a pro-forum bias."