Empirical Legal Studies

"Anecdotes," Data, and Research Design

As I have neither an interest in nor inclination to engage substantively with an on-going scholarly "dust-up," I am content to let two papers that recently "crossed my desk" speak for themselves (compare Church & Williams with Baker & Bradt). That said, and independent of this exchange in particular and multidistrict litigation in general, a few points raised by Baker (Texas) and Bradt (Berkeley), in Anecdotes Versus Data in the Search for Truth about Multidistrict Litigation, have broader applicability.

One goal of empirical research, generally stated, is to generate results that are persuasively and adequately scaffolded by the underlying data and research design. Whether a "convenience sample" eliciting self-reports from an on-line survey can reflect an unbiased, random, and representative draw from an underlying population universe is not a given and requires further analyses. That is, for such a data-generating strategy the specter of selection bias lurks and must be addressed.

Moreover, even if one is comfortable that neither selection bias nor other data shortcomings pose methodological problems, additional research design issues persist. To be more specific, and as Baker and Bradt note, if a research question seeks to meaningfully engage with, e.g., plaintiffs' satisfaction with MDL litigation experience, data on respondents' (that is, plaintiffs in MDL litigation) satisfaction (or dissatisfaction) should be compared with an otherwise equally representative and unbiased sample of plaintiffs from non-MDL cases. Without an appropriate control group, the results lack a necessary reference point that facilitates interpreting the results.

Lower Fedreal Courts and Religious Claims: An Update

The Supreme Court's ongoing readjustment of long-standing Free Exercise and Establishment clause tensions has renewed scholarly attention to courts' resolutions of religious claims. And in a recent paper, Trump's Lower-Court Judges and Religion: An Initial Appraisal, Stephen Choi (NYU) et al. contribute to a growing empirical literature by levering data on Trump-appointed Circuit Court judges in Free Exercise Clause cases and comparing Trump-appointed judge outcomes with those of Non-Trump Republican- and Democratic-appointed judges.

What the authors find includes that "Trump’s judicial appointments are more religious but no less qualified than other judges," and that "Trump judges are more likely to vote in favor of the plaintiff than not only Democratic judges, but also judges appointed by other Republican presidents."

Notably, however, while Trump-appointed judges vote “pro-religion” (that is, in favor of a religious plaintiff asserting a free exercise claim) more frequently than other judges, this tilt distributes unevenly across various religions. For example, and as the table below illustrates, the authors find “... different voting patterns for Islam. For cases involving Islam, Trump judges vote in favor of Muslim plaintiffs in only 19.2% of the cases compared with 34.5% for Non-Trump Republicans and 48.0% for Democrats.” The authors note that the overall pattern that emerges is “consistent with conventional wisdom: Democrats tend to protect minority religions, and Republicans tend to protect Christianity (and possibly Judaism).”

 

 

The paper's abstract follows.

“It is widely believed that President Donald Trump’s judicial appointments reflected a strategy of appeasing evangelical Christians and other religious groups that favored a more conservative, Christian judiciary, and that in pursuing this strategy Trump sacrificed quality. We explore evidence for this theory by examining the biographies and credentials of Trump’s lower court nominees and the voting records of his judicial appointments to the circuit courts in free exercise cases. We find that Trump’s appointments to the lower courts have stronger or more numerous religious affiliations, and are affiliated with the Federalist Society and the National Rifle Association at unusually high rates, but are no less well credentialed than other judges are. We also find that Trump’s appointments to the circuit court more frequently vote in favor of Christian plaintiffs, and less frequently vote in favor of Muslim plaintiffs, in free exercise cases than judges appointed by other Republican presidents and by Democratic presidents.”

Fixed Effects vs. Random Effects

While debates persist, how fixed- and random-effect estimators differ and what they seek to accomplish, however, remain somewhat muddled in the various empirical literatures. A recent discussion on StataList, however, provides a rare moment of clarity (here) by nudging us to think about fixed- and-random effects as involving different tradeoffs (or preferences) between consistency and efficiency. As one commentator helpfully notes:

"The -fe- estimator is always consistent but inefficient if -re- is the way to go.
 The -re- estimator is inconsistent if -fe- is the way to go."

Skewed Data, Histograms & Axis "Breaks"

While skewed data distributions (e.g., a large proportion of zeros) invite an array of well-known methodological complications, they also complicated data presentation needs. For example, a standard histogram for a distribution that includes approx. 85% zeros looks something like the following:

Setting aside the methodological complications imposed by such skewed data, as it narrowly relates to descriptive data presentation needs one "dissenting" approach is to insert axis "breaks" (discussion here) that generates the following histogram:

Such a strategy may provide the optimal tradeoff between the visualization of "extreme" data and compression of the non-extreme data. This approach also has the virtue of keeping the underlying data in their raw rather than a transformed scale.