Empirical Legal Studies

The Journal of Robustness Reports

While I've had in the back of my mind for years a thought that a scholarly journal--dedicated to replication efforts--might emerge, it looks as though something like this has. Specifically, the Journal of Robustness Reports ("JRR") sets out to, not only report results on replication efforts (however defined) but, as well, on efforts to reanalyze previously published empirical findings. An excerpt of the journal's mission follows.

"The vast majority of empirical research articles report a single primary analysis outcome that is the result of a single analysis plan, executed by a single analysis team (usually the team that also designed the experiment and collected the data). However, recent many-analyst projects have demonstrated that different analysis teams generally adopt a unique approach and that there exists considerable variability in the associated conclusions. . . . In order to showcase the practical feasibility and epistemic benefits of this approach we have founded the Journal of Robustness Reports, which is dedicated to publishing short reanalyses of empirical findings. . . ."

Additional information about this open-access Journal can be found here. I have a hunch many of this blog's readers (certainly those based in law schools) might find one of the Journal's "features or bugs" intriguing. Specifically, the JRR "limits Robustness Reports to only 500 words (excluding the Acknowledgments and Disclosures section, the References section, figure/table captions, and the title page) and one display element (table/figure)." Motivation for this word count limitation includes JRR's goal to "convince mainstream empirical journals to adopt the format and arrange their own robustness reports to accompany articles of substantial interest — an investment of 500 words is minimal and similar to that of a comment or a letter to the editor (which which many journals are already familiar)."

While I remain curious about how scholarly demand for such an outlet might unfold, I certainly hope JRR succeeds as a journal and applaud such an initiative.

Experimental Data on Insurance Contracts & Consumer Comprehension

While insurance contracts are a distinct sub-species of general contracts and subject to some insurance doctrine-specific interpretative canons, contract law generally assumes that reasonable consumers reasonably understand the contracts that they sign. Despite the uncomfortable fact that too many (most?) consumers don't even bother reading standard form contract language before signing, whether those consumers who do read contracts before signing accurately understand them endures as a separate research question.

In a recent paper, Read But Not Understood? An Empirical Analysis of Consumer Comprehension in Homeowners Insurance, Daniel Schwarcz (Minn.) et al. report results from an experiment assessing whether "providing excerpts from the dominant standard form homeowners insurance policy improves consumer understanding of coverage." Their main finding, on balance, tends to upend a pivotal contract interpretation assumption: "providing [homeowners insurance] policy language only moderately improved consumer understanding in some scenarios, while affirmatively decreasing accuracy in others." As the authors note, their findings "carry significant legal and regulatory implications."

As the experimental results derive from surveys (N=2,440) and lever a large on-line survey platform (Dynata), important external validity questions linger. Methodological questions and limitations notwithstanding, the paper's core results succeed in drawing attention to core insurance contract assumptions. An excerpted abstract follows.

"Through a series of survey-based experiments, we compare consumers’ general beliefs about homeowners insurance with their beliefs when provided with key policy excerpts. Our main finding is that providing policy language only moderately improved consumer understanding in some scenarios, while affirmatively decreasing accuracy in others. Respondents often struggled with partial reading or misinterpretation of policy provisions, especially when broad coverage grants were later restricted by specific exclusions—a common structural feature of insurance policies."

Pairwise Mean Comparison Command (pwmc)

An updated version of a command -- pwmc -- is now available for Stata users via SSC and is discussed (and described more fully) on Statalist (here). The command performs pairwise comparisons of means with unequal variances.

As the discussion makes clear, the pwmc command differs from other, related commands with the mcompare() option, such as margins with the pwcompare option, as it (pwmc) does not assume equal variances where the alternatives typically do.