Empirical Legal Studies

Sentencing Mitigation Data

As guilty pleas continue to largely define the criminal adjudication context, given judges' latitude when it comes to sentencing a deeper understanding of how mitigation efforts might vary across defendants becomes important. While prior empirical work on this question implies that "mitigation plays an important role in determining sentences," comparatively under explored is how this "role" distributes.

In a recent paper, Equality in Sentencing Mitigation, John Meixner Jr. (Georgia) brings helpful data to this question. Specifically, the paper randomly draws a weighted sample (N=302) of federal felony cases from across all federal districts between 2015 and 2018. These cases were then matched the the defense counsels' "sentencing memorandum" data from PACER as well as case-level background data from the US Sentencing Commission.

The paper's core findings include: 1) "an additional 1,000 words of mitigation about a defendant’s personal circumstances predicts an approximately 13% reduction in sentence;" and 2) as Table 5 (below) implies, the paper does "not find evidence that race or politics have a substantial impact on the effectiveness of mitigation." An excerpted abstract follows.

 "... In this Article, I present new data that shed light on these questions for the first time. I do this by leveraging two unique datasets. First, by systematically examining over 350 federal sentencing memoranda and coding them for the length of arguments devoted to various categories of mitigation (such as arguments about defendants’ good character, traumatic upbringing, or history of mental or physical illness). ... The results are encouraging. For the most part, I do not find evidence that race or politics have a substantial impact on the effectiveness of mitigation. In that way, mitigation may function as a sort of rising tide that lifts all boats, aiding defendants regardless of characteristics that may result in inequality in other parts of criminal procedure."

State FE/ME Discussion

Insofar as a lot of ELS engages with variation across states, some may be interested in this recent--and growing--listserv discussion (click here) on various ways to handle states as a variable (either as a control or treatment).

Organ Donor Registrations and Research Designs

For years, health policymakers have struggled with reconciling a gap (real or perceived) between people who voice public support for organ donation (90.4%) and those who actually agree to serve as organ donors (49.9%). In this policy space, two main organ donor approaches dominate ("yes/no" and "opt-in") and for years researchers have wondered "whether how individuals are asked to become organ donors might impact registration rates." Prevailing wisdom largely implies that because "the opt-in frame emphasizes that the default is not being a donor, which individuals might construe as an implicit recommendation of policymakers." Thus, as states moved from an "opt-in" to a "yes/no" approach to potential organ donors, researchers began to assess, primarily through survey experiments, whether "the yes/no question frame actually increase organ donor registrations."

In a recent paper, Increasing Organ Donor Registration as a Means to Increase Transplantation: An Experiment with Actual Organ Donor Registrations, the authors, Judd B. Kessler (Penn-Wharton) & Alvin E. Roth (Stanford--econ.), lever a “field-in-the-lab” research design and find, in contrast with data drawn from hypothetical donation decision experiments, no statistical difference between the two competing donor approaches. The paper's excerpt follows.

"The United States has a severe shortage of organs for transplant. Recently—inspired by research based on hypothetical choices—jurisdictions have tried to increase organ donor registrations by changing how the registration question is asked. We evaluate these changes with a novel “field-in-the-lab” experiment, in which subjects change their real organ donor status, and with new donor registration data collected from US states. A “yes/no” frame is not more effective than an “opt-in” frame, contradicting conclusions based on hypothetical choices, but other question wording can matter, and asking individuals to reconsider their donor status increases registrations."

CELS 2025--Call For papers

At the request of the CELS 2025 conference co-organizers, David Hyman, Neel Sukhatme, and Josh Teitelbaum, I am delighted to pass along the following information:

Conference on Empirical Legal Studies

24-25 October 2025

Call For papers

The 2025 Conference on Empirical Legal Studies (CELS) is now accepting submissions. CELS will consider empirical papers spanning all areas of empirical legal studies. Authors are encouraged to submit works-in-progress; however, submissions should be completed drafts that include principal results. Submitted papers must be unpublished (and expected to be unpublished at the time of the conference).

The paper submission deadline is Friday, June 20, 2025 (11:59 PM Eastern Daylight Time). There is no charge for submissions.

To submit a paper, please visit www.cels2025.com. We encourage preregistration of all experiments, and we encourage all authors to make their data, scripts, and code publicly available upon publication to the extent your methodology permits. We welcome replication studies and papers that develop and demonstrate empirical methods.

Papers are selected through a peer-review process. If accepted, authors will have an opportunity to submit a revised draft prior to the conference for presentation and discussion. Discussion at the conference includes assigned commentators and audience questions. Accepted papers will be made available to all conference participants. Acceptance of a paper or poster constitutes the presenting author’s agreement to be a commentator on another paper at the conference.

The 19th Annual Conference on Empirical Legal Studies (CELS) will be held on 24-25 October 2025, at Georgetown University Law Center in Washington, D.C. For more conference information click here; conference organizers can be contacted at: [email protected].

CELS 2025 is co-sponsored by Georgetown Law and the SELS.

The Journal of Robustness Reports

While I've had in the back of my mind for years a thought that a scholarly journal--dedicated to replication efforts--might emerge, it looks as though something like this has. Specifically, the Journal of Robustness Reports ("JRR") sets out to, not only report results on replication efforts (however defined) but, as well, on efforts to reanalyze previously published empirical findings. An excerpt of the journal's mission follows.

"The vast majority of empirical research articles report a single primary analysis outcome that is the result of a single analysis plan, executed by a single analysis team (usually the team that also designed the experiment and collected the data). However, recent many-analyst projects have demonstrated that different analysis teams generally adopt a unique approach and that there exists considerable variability in the associated conclusions. . . . In order to showcase the practical feasibility and epistemic benefits of this approach we have founded the Journal of Robustness Reports, which is dedicated to publishing short reanalyses of empirical findings. . . ."

Additional information about this open-access Journal can be found here. I have a hunch many of this blog's readers (certainly those based in law schools) might find one of the Journal's "features or bugs" intriguing. Specifically, the JRR "limits Robustness Reports to only 500 words (excluding the Acknowledgments and Disclosures section, the References section, figure/table captions, and the title page) and one display element (table/figure)." Motivation for this word count limitation includes JRR's goal to "convince mainstream empirical journals to adopt the format and arrange their own robustness reports to accompany articles of substantial interest — an investment of 500 words is minimal and similar to that of a comment or a letter to the editor (which which many journals are already familiar)."

While I remain curious about how scholarly demand for such an outlet might unfold, I certainly hope JRR succeeds as a journal and applaud such an initiative.

Experimental Data on Insurance Contracts & Consumer Comprehension

While insurance contracts are a distinct sub-species of general contracts and subject to some insurance doctrine-specific interpretative canons, contract law generally assumes that reasonable consumers reasonably understand the contracts that they sign. Despite the uncomfortable fact that too many (most?) consumers don't even bother reading standard form contract language before signing, whether those consumers who do read contracts before signing accurately understand them endures as a separate research question.

In a recent paper, Read But Not Understood? An Empirical Analysis of Consumer Comprehension in Homeowners Insurance, Daniel Schwarcz (Minn.) et al. report results from an experiment assessing whether "providing excerpts from the dominant standard form homeowners insurance policy improves consumer understanding of coverage." Their main finding, on balance, tends to upend a pivotal contract interpretation assumption: "providing [homeowners insurance] policy language only moderately improved consumer understanding in some scenarios, while affirmatively decreasing accuracy in others." As the authors note, their findings "carry significant legal and regulatory implications."

As the experimental results derive from surveys (N=2,440) and lever a large on-line survey platform (Dynata), important external validity questions linger. Methodological questions and limitations notwithstanding, the paper's core results succeed in drawing attention to core insurance contract assumptions. An excerpted abstract follows.

"Through a series of survey-based experiments, we compare consumers’ general beliefs about homeowners insurance with their beliefs when provided with key policy excerpts. Our main finding is that providing policy language only moderately improved consumer understanding in some scenarios, while affirmatively decreasing accuracy in others. Respondents often struggled with partial reading or misinterpretation of policy provisions, especially when broad coverage grants were later restricted by specific exclusions—a common structural feature of insurance policies."

Pairwise Mean Comparison Command (pwmc)

An updated version of a command -- pwmc -- is now available for Stata users via SSC and is discussed (and described more fully) on Statalist (here). The command performs pairwise comparisons of means with unequal variances.

As the discussion makes clear, the pwmc command differs from other, related commands with the mcompare() option, such as margins with the pwcompare option, as it (pwmc) does not assume equal variances where the alternatives typically do.

Experimental Designs with Unequal Variance Across Treated and Control Data

While I generally do not dabble all that much in experiments, I am familiar enough with the literature to know that, aside from general statistical power concerns, ex ante questions about how many subjects should be in the treated and control groups elude easy answers. One variant to this question considers when a researcher should have unequal numbers of units in the treatment and control groups. To be sure, such practical factors as a treatment's cost and/or disruption surely play some role. A recent discussion on Andrew Gelman's blog (here) takes up more complicated issues relating to how best to maximize the statistical efficiency of treated and control subjects' distribution and describes how simulations might assist.

Northwestern Univ. L. Rev.'s 7th Annual "Empirical Issue": Call For Papers

If evidence of empirical legal studies’ growth was not already obvious enough in the peer-reviewed world, some movement in traditional student law reviews also evidences a turn toward ELS as well. And no more so than perhaps the Northwestern University Law Review which is pleased to announce its seventh annual issue dedicated to empirical legal scholarship, to be published in Spring 2026. The student editors "seek to bring cutting-edge, interdisciplinary, empirical work to our legal audience, and enrich our understanding of the law, legal actors, and legal doctrine through robust and reliable examination of qualitative, quantitative, and mixed-method data."

The student editors note that the "exclusive submission window for the Vol. 120 Empirical Issue of the Northwestern University Law Review will open on April 1, 2025, and run until April 28, 2025. A subset of submissions will be selected to move forward to peer review. The Law Review promises make every effort to notify authors of rejection or of advancement to peer review by June 30, 2025. Final publication decisions will be issued by August 10, 2025." Submissions must land in the 15,000-30,000 word count range.

For those interested, additional details are found here.

CELS 2025--Save the Date

At the request of the CELS 2025 conference co-organizers, David Hyman, Neel Sukhatme, and Josh Teitelbaum, I am delighted to pass along the following information:

Conference on Empirical Legal Studies

24-25 October 2025

 

The 19th Annual Conference on Empirical Legal Studies (CELS) will be held on 24-25 October 2025, at Georgetown University Law Center in Washington, D.C.

 

More conference information here.

 

CELS 2025 is co-sponsored by Georgetown Law and the SELS.

Public Input Data on SEC Rulemaking

Incident to its rulemaking activities, the SEC, similar to other agencies, solicits public input on proposed rules and reduces to formal memoranda SEC meetings with industry advocates (and lobbyists) seeking to influence final rules. The emergence of large language models (LLMs) enables scholars to assess possible relations between both forms of public input (comments and memoranda) and final rules. And this is precisely what Adam Pritchard (Michigan) et al., set out to do in their recent paper, SEC Rulemaking: An Empirical Analysis of Comments and Memoranda.

The paper draws from 482 rules, 1018 releases, 84,828 comments, and 5,700 memoranda, which are then linked (through Westlaw and Lexis) to any subsequent trial and appellate court activity. These data were then analyzed for "sentiment" with leading LLMs, including OpenAI, specifically GPT-3.5. A focus on rulemaking activity, at least with respect to SEC's proposed rules, is warranted owing to, at the very least, a detectable trend towards increased public commentary over time as the figure (below) implies.

Among this paper's key findings include that while comment volume and sentiment systematically inform final SEC rules, unsurprisingly, this influence varies across different types of commentators. Similarly, direct advocacy engagement with SEC officials--indicated by SEC memoranda--"is an important marker of proposed rules’ adoption, and if adopted, judicial challenge and survival." An excerpted abstract follows.

"... We find that public inputs contain valuable information regarding the probability that a rule is adopted, challenged, and survives judicial scrutiny. Longer average comments and greater disagreement among commenters correlate with a lower likelihood of rule survival, whereas an increase in the number of suggestions correlates with a greater survival likelihood. Rules affecting issuers are less likely to be adopted and more likely to face a court challenge, while rules affecting broker-dealers are more likely to survive judicial scrutiny. As for specific issues, comments emphasizing the burden on competition correlate with a lower likelihood of rule survival, comments raising constitutional questions reduce both the likelihood of rule adoption and survival conditional on a judicial challenge. Comments by academics have no statistically significant effect at any stage of the process...."

14th Annual Workshop on Research Design for Causal Inference

Once again hosted at Northwestern's Pritzker School of Law, Chicago, IL, the 14th annual summer Workshop on Research Design for Causal Inference will include distinct--though related--"Main" and "Advanced" workshop sessions.

Main Workshop:  Monday – Friday, July 28-August 1, 2025; Advanced Workshop:  Sunday – Wednesday, August 3-6, 2025.

Main Workshop Topics/Faculty: Introduction to Modern Methods for Causal Inference:  Donald B. Rubin (Harvard), Matching and Reweighting Designs for "Pure" Observational Studies: Jens Hainmueller (Stanford), Panel Data and Difference-in-Differences: Jens Hainmueller (Stanford), Regression Discontinuity: Heather Royer (UCSB), and Instrumental Variable Methods: Tymon Sloczynski (Brandeis).

Advanced Workshop Topics/Faculty: Primer on machine learning approaches to prediction: Christian Hansen (Chicago), Applications of machine learning to causal inference:  Christian Hansen (Chicago), Advanced Difference-in Differences: Andrew Goodman-Bacon (Fed. Reserve Bd.), and Advanced Instrumental Variables: Peter Hull (Brown).

In person-registration is limited to 125 participants for each workshop (there will also be a Zoom option as well).

For further information and registration: (click here).

Binary Choice Models: Logit/Probit/LPM/LDM?

In the "old days," debates about binary choice models (where the dependent variable= [0,1]) typically centered on probit v. logit. More recent discussions of binary choice models, however, have broadened to include various linear regression possibilities, including LPM and LDM models (click here for one recent example). As discussions of binary choice model options continue to evolve--and some of these discussions evidence (or reflect) variation in norms across fields (e.g., poli sci, econ, epi, etc.)--so to do resources explaining the competing costs and benefits of various empirical strategies. For examples of a few helpful resources, click here and here.