Empirical Legal Studies

The Journal of Robustness Reports

While I've had in the back of my mind for years a thought that a scholarly journal--dedicated to replication efforts--might emerge, it looks as though something like this has. Specifically, the Journal of Robustness Reports ("JRR") sets out to, not only report results on replication efforts (however defined) but, as well, on efforts to reanalyze previously published empirical findings. An excerpt of the journal's mission follows.

"The vast majority of empirical research articles report a single primary analysis outcome that is the result of a single analysis plan, executed by a single analysis team (usually the team that also designed the experiment and collected the data). However, recent many-analyst projects have demonstrated that different analysis teams generally adopt a unique approach and that there exists considerable variability in the associated conclusions. . . . In order to showcase the practical feasibility and epistemic benefits of this approach we have founded the Journal of Robustness Reports, which is dedicated to publishing short reanalyses of empirical findings. . . ."

Additional information about this open-access Journal can be found here. I have a hunch many of this blog's readers (certainly those based in law schools) might find one of the Journal's "features or bugs" intriguing. Specifically, the JRR "limits Robustness Reports to only 500 words (excluding the Acknowledgments and Disclosures section, the References section, figure/table captions, and the title page) and one display element (table/figure)." Motivation for this word count limitation includes JRR's goal to "convince mainstream empirical journals to adopt the format and arrange their own robustness reports to accompany articles of substantial interest — an investment of 500 words is minimal and similar to that of a comment or a letter to the editor (which which many journals are already familiar)."

While I remain curious about how scholarly demand for such an outlet might unfold, I certainly hope JRR succeeds as a journal and applaud such an initiative.

Experimental Data on Insurance Contracts & Consumer Comprehension

While insurance contracts are a distinct sub-species of general contracts and subject to some insurance doctrine-specific interpretative canons, contract law generally assumes that reasonable consumers reasonably understand the contracts that they sign. Despite the uncomfortable fact that too many (most?) consumers don't even bother reading standard form contract language before signing, whether those consumers who do read contracts before signing accurately understand them endures as a separate research question.

In a recent paper, Read But Not Understood? An Empirical Analysis of Consumer Comprehension in Homeowners Insurance, Daniel Schwarcz (Minn.) et al. report results from an experiment assessing whether "providing excerpts from the dominant standard form homeowners insurance policy improves consumer understanding of coverage." Their main finding, on balance, tends to upend a pivotal contract interpretation assumption: "providing [homeowners insurance] policy language only moderately improved consumer understanding in some scenarios, while affirmatively decreasing accuracy in others." As the authors note, their findings "carry significant legal and regulatory implications."

As the experimental results derive from surveys (N=2,440) and lever a large on-line survey platform (Dynata), important external validity questions linger. Methodological questions and limitations notwithstanding, the paper's core results succeed in drawing attention to core insurance contract assumptions. An excerpted abstract follows.

"Through a series of survey-based experiments, we compare consumers’ general beliefs about homeowners insurance with their beliefs when provided with key policy excerpts. Our main finding is that providing policy language only moderately improved consumer understanding in some scenarios, while affirmatively decreasing accuracy in others. Respondents often struggled with partial reading or misinterpretation of policy provisions, especially when broad coverage grants were later restricted by specific exclusions—a common structural feature of insurance policies."

Pairwise Mean Comparison Command (pwmc)

An updated version of a command -- pwmc -- is now available for Stata users via SSC and is discussed (and described more fully) on Statalist (here). The command performs pairwise comparisons of means with unequal variances.

As the discussion makes clear, the pwmc command differs from other, related commands with the mcompare() option, such as margins with the pwcompare option, as it (pwmc) does not assume equal variances where the alternatives typically do.

Experimental Designs with Unequal Variance Across Treated and Control Data

While I generally do not dabble all that much in experiments, I am familiar enough with the literature to know that, aside from general statistical power concerns, ex ante questions about how many subjects should be in the treated and control groups elude easy answers. One variant to this question considers when a researcher should have unequal numbers of units in the treatment and control groups. To be sure, such practical factors as a treatment's cost and/or disruption surely play some role. A recent discussion on Andrew Gelman's blog (here) takes up more complicated issues relating to how best to maximize the statistical efficiency of treated and control subjects' distribution and describes how simulations might assist.

Northwestern Univ. L. Rev.'s 7th Annual "Empirical Issue": Call For Papers

If evidence of empirical legal studies’ growth was not already obvious enough in the peer-reviewed world, some movement in traditional student law reviews also evidences a turn toward ELS as well. And no more so than perhaps the Northwestern University Law Review which is pleased to announce its seventh annual issue dedicated to empirical legal scholarship, to be published in Spring 2026. The student editors "seek to bring cutting-edge, interdisciplinary, empirical work to our legal audience, and enrich our understanding of the law, legal actors, and legal doctrine through robust and reliable examination of qualitative, quantitative, and mixed-method data."

The student editors note that the "exclusive submission window for the Vol. 120 Empirical Issue of the Northwestern University Law Review will open on April 1, 2025, and run until April 28, 2025. A subset of submissions will be selected to move forward to peer review. The Law Review promises make every effort to notify authors of rejection or of advancement to peer review by June 30, 2025. Final publication decisions will be issued by August 10, 2025." Submissions must land in the 15,000-30,000 word count range.

For those interested, additional details are found here.

CELS 2025--Save the Date

At the request of the CELS 2025 conference co-organizers, David Hyman, Neel Sukhatme, and Josh Teitelbaum, I am delighted to pass along the following information:

Conference on Empirical Legal Studies

24-25 October 2025

 

The 19th Annual Conference on Empirical Legal Studies (CELS) will be held on 24-25 October 2025, at Georgetown University Law Center in Washington, D.C.

 

More conference information here.

 

CELS 2025 is co-sponsored by Georgetown Law and the SELS.

Public Input Data on SEC Rulemaking

Incident to its rulemaking activities, the SEC, similar to other agencies, solicits public input on proposed rules and reduces to formal memoranda SEC meetings with industry advocates (and lobbyists) seeking to influence final rules. The emergence of large language models (LLMs) enables scholars to assess possible relations between both forms of public input (comments and memoranda) and final rules. And this is precisely what Adam Pritchard (Michigan) et al., set out to do in their recent paper, SEC Rulemaking: An Empirical Analysis of Comments and Memoranda.

The paper draws from 482 rules, 1018 releases, 84,828 comments, and 5,700 memoranda, which are then linked (through Westlaw and Lexis) to any subsequent trial and appellate court activity. These data were then analyzed for "sentiment" with leading LLMs, including OpenAI, specifically GPT-3.5. A focus on rulemaking activity, at least with respect to SEC's proposed rules, is warranted owing to, at the very least, a detectable trend towards increased public commentary over time as the figure (below) implies.

Among this paper's key findings include that while comment volume and sentiment systematically inform final SEC rules, unsurprisingly, this influence varies across different types of commentators. Similarly, direct advocacy engagement with SEC officials--indicated by SEC memoranda--"is an important marker of proposed rules’ adoption, and if adopted, judicial challenge and survival." An excerpted abstract follows.

"... We find that public inputs contain valuable information regarding the probability that a rule is adopted, challenged, and survives judicial scrutiny. Longer average comments and greater disagreement among commenters correlate with a lower likelihood of rule survival, whereas an increase in the number of suggestions correlates with a greater survival likelihood. Rules affecting issuers are less likely to be adopted and more likely to face a court challenge, while rules affecting broker-dealers are more likely to survive judicial scrutiny. As for specific issues, comments emphasizing the burden on competition correlate with a lower likelihood of rule survival, comments raising constitutional questions reduce both the likelihood of rule adoption and survival conditional on a judicial challenge. Comments by academics have no statistically significant effect at any stage of the process...."

14th Annual Workshop on Research Design for Causal Inference

Once again hosted at Northwestern's Pritzker School of Law, Chicago, IL, the 14th annual summer Workshop on Research Design for Causal Inference will include distinct--though related--"Main" and "Advanced" workshop sessions.

Main Workshop:  Monday – Friday, July 28-August 1, 2025; Advanced Workshop:  Sunday – Wednesday, August 3-6, 2025.

Main Workshop Topics/Faculty: Introduction to Modern Methods for Causal Inference:  Donald B. Rubin (Harvard), Matching and Reweighting Designs for "Pure" Observational Studies: Jens Hainmueller (Stanford), Panel Data and Difference-in-Differences: Jens Hainmueller (Stanford), Regression Discontinuity: Heather Royer (UCSB), and Instrumental Variable Methods: Tymon Sloczynski (Brandeis).

Advanced Workshop Topics/Faculty: Primer on machine learning approaches to prediction: Christian Hansen (Chicago), Applications of machine learning to causal inference:  Christian Hansen (Chicago), Advanced Difference-in Differences: Andrew Goodman-Bacon (Fed. Reserve Bd.), and Advanced Instrumental Variables: Peter Hull (Brown).

In person-registration is limited to 125 participants for each workshop (there will also be a Zoom option as well).

For further information and registration: (click here).

Binary Choice Models: Logit/Probit/LPM/LDM?

In the "old days," debates about binary choice models (where the dependent variable= [0,1]) typically centered on probit v. logit. More recent discussions of binary choice models, however, have broadened to include various linear regression possibilities, including LPM and LDM models (click here for one recent example). As discussions of binary choice model options continue to evolve--and some of these discussions evidence (or reflect) variation in norms across fields (e.g., poli sci, econ, epi, etc.)--so to do resources explaining the competing costs and benefits of various empirical strategies. For examples of a few helpful resources, click here and here.

2025 ICPSR Summer Quant. Methods Programs/Workshops

While best known as a leading social science data archive, one of the ICPSR's under-appreciated assets includes its suite of summer programs/workshops. For more than 50 years, the ICPSR has conducted a range (90+) of courses/seminars/workshops, principally based at the University of Michigan-Ann Arbor campus, that spans the arc of quantitative methods as well as a diverse array of social scientific fields. While I recommend in-person programs where possible, online options also exist. Workshop formats vary as well, ranging from 3-5 day courses to 4-week offerings. These offerings can be invaluable for graduate (and undergraduate) students as well as faculty. For those interested, links follow regarding additional registration (click here) and scholarship (click here) information.

Selection Effects & Prof. George Priest (1947-2024)

A prior post noting Prof. George Priest's passing predictably emphasizes his enduring influence on the topic of selection effects in litigation. Among the many papers scheduled for publication in 42:3 Yale J. on Reg. (2025) and presented at a symposium (or 'Festschrift') just months before Priest's recent passing includes 'The Selection of Disputes' at Forty, by Daniel Klerman (USC) & Yoon-Ho Alex Lee (Northwestern). The Klerman & Lee paper (abstract below) complements, contributes to, and reflects Priest's enormous scholarly legacy.

“Priest and Klein's ‘Selection of Disputes for Litigation’ has been one of the most influential law review articles of all time. This Article reviews its contribution to legal scholarship. Priest and Klein's central and enduring contribution is the recognition that some cases are more likely to settle than others. It follows that litigated disputes are not a random sample of all disputes. This basic insight is true under nearly all litigation models and is also confirmed by a large body of empirical evidence. Priest and Klein's article is also famous for its prediction that, under certain conditions, the plaintiff trial win rate will tend to approach fifty percent. That prediction, however, is not supported by most other litigation models and has received only modest support from the empirical literature. Our citation analysis also suggests that 'The Selection of Disputes for Litigation' is one of the rare articles whose importance was both recognized almost immediately and whose influence has continued to grow over several decades.”

Contingent Fees & Access to Justice: Data From NYC

While "contingent fees are certainly not without their problems," recent analyses of NYC data imply that they contribute to increased equity in terms of access to justice "between rich and poor and between whites and racial minorities."

It turns out that since 1957, New York courts require contingent fee lawyers to file a "closing statement" for all cases involving "personal injury, property damage, or wrongful death." While typically confidential, these closing statements also include settlement amounts.

In Contingent Fees and Access to Justice, the authors, Eric Helland (Claremont McKenna--econ) and Dan Klerman (USC), reports results from their analyses of closing data "from 2003–2013 from the First Appellate Division, which has supervisory authority over lawyers in Manhattan and the Bronx."

To be sure, even these rich data (especially the settlement data) are not without limitations. After all, the data only involve eleven years, two NYC boroughs, and certain tort claims. Any data limitations aside, key findings include that "the poor make claims and recover at higher rates than the rich, and African Americans and Hispanics make claims at higher rates than whites." The paper's abstract follows.

"Using a unique dataset from New York City, this article shows that contingent fees seem to have, at least partly, solved the access-to-justice problem in tort litigation. People in poorer zip codes, in fact, make legal claims at a higher rate than those in wealthier areas. This greater propensity to seek and achieve legal redress is partly explained by the fact that poor people are more likely to be injured. In addition, the disparity between rich and poor manifests itself solely in smaller claims. For larger claims, rich and poor make claims at roughly the same rate. African Americans and Hispanics also make claims at a higher rate than whites, and this difference also attenuates when one controls for accident rates or focuses on larger claims."

Transforming "Vertical" Regression Output Into a "Horizontal" Table Format

A recent Statalist discussion (here) walks readers through the coding necessary to transform the following (traditionally "vertical") regression output:

-----------------------------------------
                               mpg       
-----------------------------------------
Weight (lbs.)                      -0.006
                       [-0.009    -0.003]
                                     0.00
Displacement (cu. in.)              0.006
                       [-0.014     0.025]
                                     0.57
Turn circle (ft.)                  -0.139
                       [-0.496     0.219]
                                     0.44
Intercept                          43.588
                       [33.688    53.489]
                                     0.00
Number of observations                 74
-----------------------------------------

into a "horizontal" output format:

-------------------------------------------------------------
                       Coefficient       95% CI       p-value
-------------------------------------------------------------
Weight (lbs.)               -0.006 [-0.009    -0.003]    0.00
Displacement (cu. in.)       0.006 [-0.014     0.025]    0.57
Turn circle (ft.)           -0.139 [-0.496     0.219]    0.44
Intercept                   43.588 [33.688    53.489]    0.00
-------------------------------------------------------------