Empirical Legal Studies

Pairwise Mean Comparison Command (pwmc)

An updated version of a command -- pwmc -- is now available for Stata users via SSC and is discussed (and described more fully) on Statalist (here). The command performs pairwise comparisons of means with unequal variances.

As the discussion makes clear, the pwmc command differs from other, related commands with the mcompare() option, such as margins with the pwcompare option, as it (pwmc) does not assume equal variances where the alternatives typically do.

Experimental Designs with Unequal Variance Across Treated and Control Data

While I generally do not dabble all that much in experiments, I am familiar enough with the literature to know that, aside from general statistical power concerns, ex ante questions about how many subjects should be in the treated and control groups elude easy answers. One variant to this question considers when a researcher should have unequal numbers of units in the treatment and control groups. To be sure, such practical factors as a treatment's cost and/or disruption surely play some role. A recent discussion on Andrew Gelman's blog (here) takes up more complicated issues relating to how best to maximize the statistical efficiency of treated and control subjects' distribution and describes how simulations might assist.

Binary Choice Models: Logit/Probit/LPM/LDM?

In the "old days," debates about binary choice models (where the dependent variable= [0,1]) typically centered on probit v. logit. More recent discussions of binary choice models, however, have broadened to include various linear regression possibilities, including LPM and LDM models (click here for one recent example). As discussions of binary choice model options continue to evolve--and some of these discussions evidence (or reflect) variation in norms across fields (e.g., poli sci, econ, epi, etc.)--so to do resources explaining the competing costs and benefits of various empirical strategies. For examples of a few helpful resources, click here and here.

Options for Small and Unbalanced ("Rare Events") Data Sets

While relogit is a standard "go-to" Stata command when a data set includes "rare events," a threat of bias is enhanced if rare events derive from a "small sample." While bootstraping may be an option for some, a recent Stata listserv discussion (here) emphasizes a pair of commands -- firthlogit and exlogit -- that might make sense for others. Relatedly, Prof. Williams (ND-sociology) helpfully delves into the challenge a bit more comprehensively (here).

"T-Testing" Group Differences at Various Percentiles

While the standard t-test compares means, somewhat less clear is what to do if one wants to test, e.g., whether group differences at various percentiles achieve statistical significance. In a recent Statalist discussion (click here), the specific question posed involves an effort to determine whether “Black-White wealth differences at p10 (10^th percentile) and p90 (90^th percentile)” systematically vary.

Possible options, described in the responses, range from using the qreg and ttest commands.

Taxing Autonomy and Budget Volatility

While complaints from public entities about the policy challenges and harms incident to fiscal uncertainty attributable to budget volatility emerge on an almost annual basis, factors that inform budget volatility remain understudied. Moreover, the particular role that a public entity's taxing autonomy might play remains especially obscure in the empirical literature. However, a recent paper, Does Taxing Authority Help School Districts in Managing Revenue Volatility?, by Laiyang Ke (Ark. St. Univ.--Poli Sci), makes a helpful contribution.

The paper defines taxing autonomy in terms of the degree to which a subnational government (here, public school districts) controls their tax revenues. "Budget volatility" is understood as a function of a school district's "fluctuations and uncertainty in revenue streams and expenditure patterns." Notably, both concepts have particular traction in the US public school district context. First, school districts vary in terms of their taxing autonomy. Second, "previous research has shown that state and local government revenue volatility could lead to expenditure volatility, and uncertain spending fluctuations in school districts may impair education outcomes."

One of the paper's key contributions includes its proposed measure of school district taxing autonomy which draws from school district financial data from the 1998/1999 to 2018/2019 school years. The paper's two-pronged empirical execution features a correlated random effects model as well as D-I-D estimations. The paper's findings emphasize that "school districts with a higher degree of tax autonomy—specifically, those able to change tax rates—experience less current expenditure volatility than districts with lower degrees of tax autonomy." The abstract follows.

"This study examines the impact of tax autonomy on school district expenditure volatility in the U.S., where school districts differ in their ability to levy and adjust taxes, with some having full taxing authority and others relying on fixed appropriations or lacking control over tax rates. This research introduces a comprehensive index to measure taxing autonomy across districts and develops an ordinal classification based on this framework. Using finance data from 1998/1999 to 2018/2019 and difference-in-differences, the study investigates whether districts with higher tax autonomy experience lower expenditure volatility compared to those with limited or no tax authority. The findings reveal that districts with greater tax autonomy, particularly those able to adjust tax rates, exhibit less volatility in current expenditures. Additionally, during the Great Recession, districts with higher autonomy faced smaller cuts in current expenditures by increasing tax revenues, which helped protect educational inputs like pupil-teacher ratios."

Serial Correlation Testing for Panel Data

When working with panel data, testing for serial correlation is common—if not expected—diagnostic. To this end, a new user-created command (v.1.0.3) – xtdpdserial – functions as a postestimation command for many (though not all) standard commands (e.g., regress, xtreg, etc.).

For those interested, installation download sites include: ssc install xtdpdserial as well as: net install xtdpdserial, from(http://www.kripfganz.de/stata/)

On a distinct, though related, point, Prof. Kripfganz's (Exeter Business School--Econ. Dept.) website includes a very helpful array of related Stata user-commands (click here).

Modeling Panel Data: More Options

While panel data bring with them important benefits, standard modeling challenges necessarily lurk. For example, one can estimate fixed-effects models to control for endogeneity, but at the potential cost of forgoing studying time-invariant variables. Alternatively, one could estimate a random-effects model, but introduce potential bias.

When confronting such a trade-off with panel data, my initial default position is to run two separate models (FE and RE) and assess whether any key results are model-specific. However, other empirical strategies have emerged relatively recently. While I am unsure whether similar coding possibilities exist in R, SAS, or SPSS, Stata's correlated random-effects model (CRE) allows one to estimate both time-varying and time-invariant effects while controlling for endogeneity that arises when variables correlate with unobserved panel-level characteristics. That is, CRE models allow researchers to, in essence, "have it both ways at the same time" by controlling for any endogeneity by including the panel means of time-varying variables as additional controls in the model. For a brief introduction and helpful examples and videos, click here, here, and here.

A Brief Introductory Primer on DiD Models

While treatment effect research designs continue to grow in popularity in various empirical literatures, too many researchers (as well as non-specialists) remain unclear about their objectives. To this end, a recent addition to Stata's instructional video archive includes a brief primer (here) on Stata's DiD commands as well as an easy-to-understand visual explication of what treatment effects models set out to accomplish. This particular video (one in a series) focuses on Stata's two core DiD commands--didregress and xtdidregress;the former command is appropriate for repeated cross-sectional data and the latter for panel data.

Bonferroni Correction & T-Tests

While typically under-appreciated, multiple comparisons typically warrant consideration of whether a Bonferroni correction is necessary to reduce the risk of generating "false-positive" results. A recent discussion in the multiple t-tests context (here) includes suggested Stata coding on how one can incorporate a Bonferroni correction in a command for a series of t-tests.

Power Testing

Too often I read papers that consider various explanations for a null result save for an obvious one: sample size. That is, a null result may simply be an artifact of a too small sample. And in some research settings, notably many experimental settings, an ex ante power calculation can inform researchers on the necessary sample size before experiments are executed. A few resources (e.g., here, here, and here) provide some general introductory context and examples.

Where "vce(cluster var)" and "vce(robust)" Differ and Where They Do Not

A recent discussion on the StataList (here) identifies something of an anomaly when it comes to how "vce()" operates for the xtreg,fe command versus other commands.

As the unfolding discussion notes, when using the xtreg,fe command, vce(cluster var) and vce(robust) operate similarly. With other commands, by contrast, they generally operate differently. The difference emerges because where vce(robust) requires that observations are independent in order for it to be valid, vce(cluster var), by contrast, accommodates and adjusts for correlation of observations within values of groupvar (or "cluster var").

Generating Lagged Variable Values

The need for lagged variables is not uncommon, particularly in the finance and statutory intervention areas or whenever the effective timing of any exogenous event of interest is unclear. How to manipulate an existing variable to generate lagged variants--as well as some common pitfalls--are discussed in a recent StataList discussion (here).

2024 Northwestern Main and Advanced Causal Inference Workshops

Co-organized by Bernie Black (Northwestern) and Scott Cunningham (Baylor—econ), the 13th annual workshop on Research Design for Causal Inference will be held at Northwestern Law School in Chicago, IL.

Main Workshop: Monday – Friday, July 29--August 2, 2024; Advanced Workshop: Sunday -- Wednesday, August 4-7, 2024.

Main Workshop Topics/Faculty: Introduction to Modern Methods for Causal Inference:  Donald B. Rubin (Harvard Univ.); Matching and Reweighting Designs for “Pure” Observational Studies (Brigham Frandsen, BYU); Panel Data and Difference-in-Differences (Yiqing Xu, Stanford); Instrumental variable methods (Eric French, Cambridge); and Regression Discontinuity (Gonzalo Vazquez-Bare, UC Santa Barbara).

Advanced Workshop Topics/Faculty: Primer on machine learning approaches to prediction and Applications of machine learning to causal inference (Christian Hansen, Chicago); Advanced Difference-in Differences (Andrew Goodman-Bacon, Federal Reserve Board); and Advanced Instrumental Variables (Peter Hull, Brown).

For further information and registration: (click here).

Various Tests for Fixed- v. Random-Effects Model Decision

When assessing whether a model warrants fixed- or random-effects, threats posed by, among other factors, autocorrelation and heteroscedasticity typically loom large. A recent discussion of this topic emerged on the StataList (click here) and considers various approaches.

A Perspective on an "Empirical Turn" in Comparative Constitutional Law

In a recent (and mercifully brief) review essay, Empirical Constitutional Studies (reviewing Adam Chilton & Mila Versteeg's How Constitutional Rights Matter (OUP, 2020)), Arthur Dyevre (KU Leuven Centre for Legal Theory and Empirical Jurisprudence) takes the occasion to opine more generally on ELS' recent (and growing) "intrusion" into the comparative constitutional law field.

When it comes to empirically assessing constitutions and constitutional rights, Dyevre is mindful of the obvious methodological challenges for empiricists. After all, "comparing countries, even when observed over many years, is difficult because each country is the abstraction of complex, intricate and often idiosyncratic social dynamics that, even in the best of circumstances, are only imperfectly tracked by available data." Moreover, even where high quality data do exist, other complications lurk. "Countries may differ in unobserved ways, which in turn may correlate with both institutional choices (adding or not a particular right to the constitution) and de facto observance of rights." Finally, empirical findings in comparative constitutional law studies are invariably based on observational data drawn from non-randomized research designs which limits opportunities for causal inference.

Despite these (and other) challenges, Dyevre's review of Chilton & Versteeg's work implies a level of optimism (if cautious) for empirical methodologies' ability to contribute to such fields as comparative constitutional law (particularly so for "mixed-methods" projects).

Nesting v. Clustering: Hospital Data

Both an interesting--and instructive (that is, the response)--brief discussion incident to a question about whether to nest or cluster health care data in a hospital setting emerged recently in the StataList. That a single patient may have been admitted into multiple hospitals over time is yet another complicating wrinkle that warrants attention. One take-away from the discussion is that every detail involving how to structure an analysis requires careful consideration.

New Thinking About an Old Problem: Selection Effects

Keith Hylton (Boston Univ.) and Sanghoon Kim (Buffalo--econ.) distill their recent paper's challenge nicely. When it comes to much of empirical work on legal (and court) outcomes, the full potential of multiple regression "has not been realized, largely because of one reason: the information contained in court opinions comes from a selected sample."

For all the obvious reasons, much of empirical legal research exploits court outcomes as data. One challenge, however, is that the comparatively smaller subset of formal legal disputes that persist to a judicial outcome almost assuredly systematically differs from the much larger subset of legal disputes that privately settle. My former mentor and colleague, Ted Eisenberg, frequently reminded me that "one cannot over-state the importance of selection effects in empirical legal research."

While the challenge endures, solutions are not entirely obvious. In Trial Selection and Estimating Damages Equations, Hylton and Kim develop an interesting suggestion. Specifically, they articulate a structural econometric model that, they argue, better accounts for litigant selections in the appellate context. The paper's abstract follows.

"Many studies have employed regression analysis with data drawn from court opinions. For example, an analyst might use regression analysis to determine the factors that explain the size of damages awards or the factors that determine the probability that the plaintiff will prevail at trial or on appeal. However, the full potential of multiple regression analysis in legal research has not been realized, largely because of the sample selection problem. We propose a method for controlling for sample selection bias using data from court opinions."

Deploying Large Language Models in the Service of Empirical Legal Studies

The release of ChatGPT, among other large language models (LLMs), materially altered the natural language processessing landscape. Questions about whether and, more to the point, how, these tools could advance empirical legal studies quickly emerged and, unsurprisingly, persist.

In a recent paper, Language Model Interpretability and Empirical Legal Studies, Michael Livermore (UVa), et al., undertake two tasks: "(1) to present an empirical comparison of LLMs and non-LLM on a sampling of fundamental tasks, and (2) to examine the use of LLMs within the norms of ELS scholarship." The paper specifically seeks to place "the current capabilities and limitations of LLMs within the larger landscape of techniques used to convert law into data." The authors report that their comparisons yielded "mixed" results, noting that existing LLMs outperform non-LLM natural language processessing techniques in some contexts but not in others. The paper's abstract follows.

"Large language models (LLMs) now perform extremely well on many natural language processing tasks. Their ability to convert legal texts to data may offer empirical legal studies (ELS) scholars a low-cost alternative to research assistants in many contexts. However, less complex computational language models, such as topic modeling and sentiment analysis, are more interpretable than LLMs. In this paper we highlight these differences by comparing LLMs with less complex models on three ELS-related tasks. Our findings suggest that ELS research will—for the time being—benefit from combining LLMs with other techniques to optimize the strengths of each approach."

Levering AI in the Service of Contract Interpretation

Contract interpretation canons and theories continue to evolve. The latest foray seeks to lever AI.

In a recent article, Generative Interpretation, the authors, David Hoffman (Penn) and Yonathan Arbel (Alabama), advance a new approach designed to assist in determining contracting parties’ meaning. The authors' impulse behind generative interpretation, is, according to them, "simple:" they apply large language models (LLMs) to contractual texts and extrinsic evidence to predict what the parties would have said at contracting about what they meant. The paper's abstract follows.

“We introduce generative interpretation, a new approach to estimating contractual meaning using large language models. As AI triumphalism is the order of the day, we proceed by way of grounded case studies, each illustrating the capabilities of these novel tools in distinct ways. Taking well-known contracts opinions, and sourcing the actual agreements that they adjudicated, we show that AI models can help factfinders ascertain ordinary meaning in context, quantify ambiguity, and fill gaps in parties’ agreements. We also illustrate how models can calculate the probative value of individual pieces of extrinsic evidence.

After offering best practices for the use of these models given their limitations, we consider their implications for judicial practice and contract theory. Using LLMs permits courts to estimate what the parties intended cheaply and accurately, and as such generative interpretation unsettles the current interpretative stalemate. Their use responds to efficiency-minded textualists and justice-oriented contextualists, who argue about whether parties will prefer cost and certainty or accuracy and fairness. Parties—and courts—would prefer a middle path, in which adjudicators strive to predict what the contract really meant, admitting just enough context to approximate reality while avoiding unguided and biased assimilation of evidence. As generative interpretation offers this possibility, we argue it can become the new workhorse of contractual interpretation.”

"Anecdotes," Data, and Research Design

As I have neither an interest in nor inclination to engage substantively with an on-going scholarly "dust-up," I am content to let two papers that recently "crossed my desk" speak for themselves (compare Church & Williams with Baker & Bradt). That said, and independent of this exchange in particular and multidistrict litigation in general, a few points raised by Baker (Texas) and Bradt (Berkeley), in Anecdotes Versus Data in the Search for Truth about Multidistrict Litigation, have broader applicability.

One goal of empirical research, generally stated, is to generate results that are persuasively and adequately scaffolded by the underlying data and research design. Whether a "convenience sample" eliciting self-reports from an on-line survey can reflect an unbiased, random, and representative draw from an underlying population universe is not a given and requires further analyses. That is, for such a data-generating strategy the specter of selection bias lurks and must be addressed.

Moreover, even if one is comfortable that neither selection bias nor other data shortcomings pose methodological problems, additional research design issues persist. To be more specific, and as Baker and Bradt note, if a research question seeks to meaningfully engage with, e.g., plaintiffs' satisfaction with MDL litigation experience, data on respondents' (that is, plaintiffs in MDL litigation) satisfaction (or dissatisfaction) should be compared with an otherwise equally representative and unbiased sample of plaintiffs from non-MDL cases. Without an appropriate control group, the results lack a necessary reference point that facilitates interpreting the results.

Fixed Effects vs. Random Effects

While debates persist, how fixed- and random-effect estimators differ and what they seek to accomplish, however, remain somewhat muddled in the various empirical literatures. A recent discussion on StataList, however, provides a rare moment of clarity (here) by nudging us to think about fixed- and-random effects as involving different tradeoffs (or preferences) between consistency and efficiency. As one commentator helpfully notes:

"The -fe- estimator is always consistent but inefficient if -re- is the way to go.
 The -re- estimator is inconsistent if -fe- is the way to go."

More Evidence of a Racialized Death Penalty

Notably, race does not emerge as a central concern in the Court’s Furman decision, now more than fifty years old. In the decades since Furman, however, empirical evidence continues to mount documenting that "racial discrimination infects the administration of the death penalty." A recent paper by Jeffrey Fagan et al. (Columbia), Getting to Death: Race and the Paths of Capital Cases after Furman, represents a recent and important contribution to this literature. Researchers visited each judicial circuit in Georgia to record the details of every murder case that entered the courts from 1994–2005. Analyses of the 1,317 death-eligible murders suggest that “race continues to matter, and that the statutory aggravators that distinguish death cases from other killings are themselves racialized.” A more detailed description of the paper's core findings appears in the Abstract (below).

“Decades of research on the administration of the death penalty have recognized the persistent arbitrariness in its implementation and the racial inequality in the selection of defendants and cases for capital punishment. This Article provides new insights into the combined effects of these two constitutional challenges. We show how these features of post-Furman capital punishment operate at each stage of adjudication, from charging death-eligible cases to plea negotiations to the selection of eligible cases for execution and ultimately to the execution itself, and how their effects combine to sustain the constitutional violations first identified 50 years ago in Furman. Analyzing a dataset of 2,328 first- degree murder convictions in Georgia from 1995–2004 that produced 1,317 death eligible cases, we show that two features of these cases combine to produce a small group of persons facing execution: victim race and gender, and a set of case-specific features that are often correlated with race. We also show that these features explain which cases progress from the initial stages of charging to a death sentence, and which are removed from death eligibility at each stage through plea negotiations. Consistent with decades of death penalty research, we also show the special focus of prosecution on cases where Black defendants murder white victims. The evidence in the Georgia records suggests a regime marred less by overbreadth in its statute than capriciousness and randomness in the decision to seek death and to seek it in a racially disparate manner. These two dimensions of capital case adjudication combine to sustain the twin failures that produce the fatal lottery that is the death penalty.”

Interpreting DID Output

While one may quibble about how the questioner designed her DID estimation, the question itself--how to understand and interpret standard DID output (in Stata)--elicited some helpful replies in a recent StataList discussion (here).

Law School Curriculum: Reform, Assessment, and Research Design

While age-old complaints persist about the basic structure of the first-year law school curriculum, few law schools have dared to experiment with meaningful reforms. Fewer still study these reforms coherently. The "Curriculum B" experiment at Georgetown Law School, as well as a recent paper by David Hyman et al. (Georgetown), Does the 1L Curriculum Make a Difference?, are important and interesting exceptions.

Since 1991, Georgetown Law School has offered an "alternative" Curriculum B along side a standard, more traditional first-year curriculum. Curriculum B was designed to provide “critical” perspectives on the law and respond to important changes in legal practice and theory in the years since the standard 1L curriculum was first developed decades earlier. At Georgetown, students elect into Curriculum B and, when demand exceeds supply, student assignment to Curriculum B is determined by lottery.

For assessment purposes, the quasi-randomized control trial aspect of student assignment to Curriculum B provides a way to overcome selection bias. That is, Hyman et al. compare student outcomes for those who opted into Curriculum B (the treatment group) with student outcomes from those students who wanted to receive the "treatment" (Curriculum B) but, due to excess demand and random selection (lottery), were instead assigned the "placebo" (that is, to the standard, traditional first-year curriculum).

Aside from careful attention to research design, a few of the paper's warrant note. When it comes to students' overall law school academic performances, for example, the authors find that "Curriculum B leads to a mean preserving contraction in students’ overall grades. At the same time, we find that it leads to diminished performance in two key business law electives (Corporations and Securities Regulation)."

Multi-Level Model "vs." Clustering

As this question (or something like it) is not an infrequent one, I found Jeff Wooldridge's response (No.2) in this StataList discussion (here) quite helpful. The bottom line is that decisions incident to estimating a multi-level model or/and clustering pivot on an array of considerations.

Simple Problems, Simple Solutions

While I should have known this years ago, better late than never.

As one becomes familiar with the broad descriptive contours of new data, the tab function is both well-known and ubiquitous. Finding analogous basic descriptive statistics for sub-samples, while certainly possible, was typically (for me, anyway) a bit cumbersome with the tab command.

A recent discussion on the StataList, however, introduced me to the simple (and code-saving) power of a related command, tabstat. As this discussion illustrates (click here), what the tab command accomplishes with multiple lines of coding and multiple output tables, the tabstat command, by contrast, accomplishes with a single line of code and a single, unified output table.

Comparing Regression Output Across Models

As I am seeing more papers trying to do so (as well as discussions about strategies on how to do so), I thought links to a recent StataList discussion as well as some web-based resources might be of general interest. The basic set-up follows:

Model A: Y = α1 + β1X1 + Σθ∙C + firm-fixed effect + year-fixed effect.

Model B: Y = α1 + β2X2 + Σθ∙C + firm-fixed effect + year-fixed effect.

Σθ∙C is list of control variables and are the same for both model A & B.

Question: How to compare the coefficients of β1 and β2?

One approach exploits a post-estimation command in Stata (suest), and then compares means with test (for a discussion and illustration, click here). Another illustrated example, courtesy of the UCLA Statistical Methods and Data Analytics site, is here.

Approaches Toward Missing Data

Missing data vex researchers, and there's no single "correct" way to deal with this not-infrequent real-world problem. A recent question posted on the StataList (here) visually illustrates what this can look like (in terms of N variation) across alternative specifications of a multilevel regression model drawing from a data set that contains missing data in some variables.

As this output illustrates, as one proceeds across models 1-4 what is clear is that the addition (and subtraction) of variables introduce missing data. How to approach this and what to do about it motivated the initial query. And in the discussion that follows identifies (some of the) various "standard" approaches.

Mistaking Correlation for Causation

Economists--and econometrics--have been involved in empirical legal studies for generations. This is particularly true for anti-trust doctrine, both its development and current application. Legal challenges to proposed mergers, and questions relating to anti-competitive spill-over effects, including market concentration, remain a staple for empirical analyses.

In the anti-trust context, one intuitive move involves assessing whether "prices are positively related to measures of concentration, such as the Herfindahl-Hirschman Index (HHI), comparing across different geographic markets or time periods." Pursuing such a question can involve (and frequently does involve) something as simple (and potentially powerful) as regressing price on the HHI.

Despite whatever intuitive appeal such a move holds, in a recent brief essay--co-signed by many leading law and economics scholars (including, and I add in interest of full disclosure, my Cornell colleague George Hay), On the Misuse of Regressions of Price on the HHI in Merger Review, the authors caution against such a move as it invites one basic fundamental problem: Mistaking correlation for causation. And it is for the essay's description of just how such a basic mistake can arise (rather than it's specific application in the anti-trust context) that I recommend this essay, especially to younger, developing empirical legal scholars.

As it relates to the specific anti-trust point developed in the essay, the authors' argue that: "The underlying problem with regressions of price on the HHI is that the relationship between price and the HHI is not causal. Instead, both are equilibrium outcomes that are determined by demand, supply, and the factors that drive them. Thus, a regression of price on the HHI does not show the sort of causal effect that would be helpful in predicting the competitive effects of a merger." More specifically, as the "HHI can take values ranging from 0 to 10,000, with the former corresponding to a fragmented market with infinitesimal firms, and the latter to monopoly. All else equal, a decrease in the number of firms leads to a higher HHI. However, the HHI can increase if, for a given number of firms, the market shares of the larger firms increase."

The root problem, of course, flows from challenges posed by “endogeneity” or “simultaneity.” Such problems lurk "when a clear causal path between two variables is posited by the relevant economic theory but the causal factor (on the right side of the regression equation) is either correlated with unobserved factors or, more subtly, is simultaneously determined by the variables (in this case, price) on the left side of the regression. A famous example of simultaneity involves price and quantity, which are simultaneously determined in the classic economic model of supply and demand." The problems posed by "endogeneity” or “simultaneity," of course, are not limited to the anti-trust realm. An excerpted abstract follows.

"It might seem natural to determine whether prices are positively related to measures of concentration, such as the Herfindahl-Hirschman Index (HHI), comparing across different geographic markets or time periods. This might be implemented by using a simple regression of price on the HHI. However, for reasons that we describe, regressions of price on the HHI should not be interpreted as establishing causation. That is, they do not inform how a change in concentration from a merger would affect prices. Courts and other policy-makers should not rely on regressions of price on the HHI for the purposes of antitrust merger review."

Writing Up Results From a Logit Model Using the Odds Ratio Option

A fairly innocuous question -- how best to interpret odds ratio results from a xtlogit model estimation -- unleashed a torrent of commentary on the StataList (click here). As this is a not uncommon model used in many ELS papers, I thought others may benefit from the (extended) discussion.

One of the (many) points that garners attention includes the numeric range for odds ratios: "There is no reason an odds ratio can't be greater than 2. On the other hand, an odds ratio can never be less than 0. The possible values for an odds ratio range from 0 to + infinity. And the interpretation does not depend on the size of the odds ratio. In your case, assuming the predictor is dichotomous, it means that the odds of a positive outcome when the predictor is 1 is 2.46 times the odds of a positive outcome when the predictor is 0. If the predictor is continuous, it means that a unit difference in the value of the predictor is associated with the odds of a positive outcome increasing by a factor of 2.46."

Logical Limits of the Randomized Controlled Trial Research Design

For many, and for obvious methodological reasons, the randomized control trial (“RCT”) is "the" (or "a") research design "gold standard" for those seeking data with possible causal implications. (For a nice quick description of RCTs click here.) In contrast, those interested in a clever, "tongue-in-cheek," "critique" of the pursuit of absolute research design "purity" might be interested in a 2003 "essay," Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomized controlled trials, by Gordon CS Smith (Cambridge--medicine) et al. In the "essay" the authors observe that they could not find any "published research presenting results from randomized controlled trials of parachute use." Consequently, the essay "argues" that the empirical basis for the parachute's efficacy is "purely observational" and, moreover, that existing "observational" data on the parachute's apparent efficacy could potentially be explained by a "healthy cohort" effect.

Marginal Effects

While journals have traditionally focused on a key variable's direction and statistical significance, over time an increasing number of journal editors (and reviewers) ask for explications of a variable's substantive and "practical" influence. This trend is unlikely to abate anytime soon. And this is precisely where marginal effects (and Stata's margins command) come into play.

While no longer a new topic, I want to pass along the following resource, Using Stata’s Margins Command to Estimate and Interpret Adjusted Predictions and Marginal Effects, by Richard Williams (Notre Dame--sociology). The document is, at once, accessible, thorough, and yet not over-bearing as it walks readers through the basics of the  margins command. The presentation is greatly enhanced by step-by-step illustrations and the necessary coding. In all, quite a helpful resource for those seeking a quick tutorial.

Testing the Difference Between Two Regression Coefficients Within Same Model

Over on the StataList, a very helpful discussion was launched by the following question that may interest folks here:

"I have one single regression model estimating the coefficients (b) of two independent variables (both are binary dummy variables). After the regression, I would like to do a t-test which calculates the difference between the two variables' coefficients (bIV1 - bIV2) and then gives a p-value for that (i.e. p-value: bIV1 = bIV2)."

To generate a Wald test (and associated p-value), all that is needed is the test command. If, however, one wants to see magnitude (and confidence interval) of the difference between two coefficients within a model, lincom is the appropriate command. For an helpful illustration of how this command works (with output), click here.

Technology and Legal Citation "Stickiness"

An interesting citation paper caught my eye, principally for its research design. In Citation Stickiness, Computer-Assisted Legal Research, and the Universe of Thinkable Thoughts, Aaron Kirschenfeld (UNC) and Alexa Chew (UNC), set out to explore how citation “stickiness” has survived the emergence of a “digital era” in legal research.

What the paper means by “stickiness” is that, in theory, the percentage of sticky citations   in   courts’   opinions   would   be   something   approaching 100%. That is, the  parties  would  discuss  the  relevant authorities in their briefs, and the court would rely  on  the  same  authorities  in  its  opinions.

Setting aside from the paper’s substantive contribution on “stickiness,” for me the paper’s main analytic move is to assess “stickiness” across dynamic legal research platforms over time. While the data are limited to a small handful of cases (N=325) from a single Circuit (CA4) and a handful of years (1957, 1987, and 2017), I anticipate that other research will employ a similar move to gain additional insights on how the emergence of the “digital era” informs law-related output. The abstract follows.

“Legal information has been available in widespread digital format for more than forty years. In that time, law librarians have wondered whether this digital switch has changed how law students and lawyers conduct research and, if so, what those changes are. Does legal research differ when conducted in print sources rather than computerized sources? What influence did the systems of organizing law in the print era have on the digital systems that followed?

While we cannot put these questions to rest, we hope to shed some light on those difference by studying the work of lawyers and courts from the print era, the transition-to-digital era, and the digital era. Using a metric called “citation stickiness,” we studied how often parties to an appeal and the judges hearing that appeal agreed on the cases relevant to resolve the issues on appeal. This, we hoped, would also show whether there were perceptible differences in coherence and stability of the legal information landscape

Citation stickiness works like this: a citation is “sticky” if it appears in at least one party’s brief and then again in the court opinion. In an initial study of 325 federal court cases from 2017, 49% of the 7,552 cases that were cited in the courts’ opinions had been cited by at least one party in a brief.

This study considers cases from the United States Court of Appeals for the Fourth Circuit decided in 1957, 1987, and 2017. After examining the citations in the briefs and opinions in our sample of cases, we compare our findings from the pre-digital era and from the digital era. What we learned surprised us.”

Adjusting for "Ambiguous" Zeros in an Ordered Logit Model

In the ELS silo, what to do with--and/or how to best interpret--"zeros" frequently persist as common challenges. In the world of standard count data, possible approaches include Zero-inflated Poisson Regression and Zero-inflated Negative Binomial Regression. Ordered logit analyses, however, require something slightly different.

To illustrate, a standard example is social science research (and explicated in this brief yet helpful video) includes, e.g., a four-siloed response to the question "How many cigarettes did you smoke each day, on average (0, 1-7, 8-12, >12)?" The possible ambiguity of the "0" response is that it may include responses from people who smoked less than one cigarette per day, on average, as well as those do not smoke cigarettes at all. And if the research question is designed to get a smokers' daily volume, the "0" response from non-smoker respondents is qualitatively different than a "0" response from smokers.

To address potential distortions to the "0" response introduced by non-smokers in this hypothetical, Stata's ziologit command fits a model for ordinal outcomes with an expected "excess" of "0" responses.

Pretrial Detention and Release Experiments

As I have an admitted soft-spot for (even quasi-) randomized control trials as a research design, a recent paper by Javier Osorio (Arizona-Govt. & Pub Pol’y) et al., Reducing Pretrial Detention: A Randomized Intervention with Public Defenders in El Salvador, caught my eye. In it, the authors exploit a "randomized intervention in El Salvador designed to improve public defenders’ capacities to secure pretrial release for defendants.”

To note that pretrial detention is a problem is to note merely the obvious. And in countries facing acute security challenges, such as El Salvador, there exists enormous pressure on criminal justice authorities to limit pretrial release. The authors contend that the overreliance on pretrial detention is driven in part by lack of information about defendants’ flight risks and the prospect of their committing further crimes while awaiting trial. And this information deficit is “the consequence of deficiencies in the public defender system, upon which economically disadvantaged citizens rely to ensure legal representation.”

To assess their hypothesis, the authors exploit an unusual (by U.S. standards, anyway) experiment involving a randomly assigned treatment to public defenders in four municipalities in El Salvador. “The treatment included specialized legal training, an improved interview protocol, and enhanced communication channels with the defendants, their relatives, and other institutions. These components sought to improve public defenders’ ability to gather evidence that would support pretrial release.”

The authors found that the treatment “increased public defenders’ requests for pretrial release by nearly 10%. The intervention also helped public defenders to secure pretrial release of their clients by 4.4%, although these results are weaker and sensitive to model specification.” The paper's abstract follows.

"Approximately one third of the global prison population is in pretrial detention, waiting for trial. Overreliance on pretrial detention exposes defendants to harsh conditions, exacerbates jail overcrowding, increases recidivism, and favors criminal governance. What policies can resource strapped countries implement to effectively address excessive pretrial detention? Based on a theoretical model focused on institutional-level efforts, we evaluate an experimental intervention implemented in El Salvador intended to increase pretrial release requests and reduce pretrial detention. The intervention randomly assigned public defenders to receive specialized legal training, an improved interview protocol, material resources, and increased communication channels. We find that this inexpensive, scalable program increased pretrial release requests from public defenders by nearly 10% (0.228 standard deviations) and increased the success in securing pretrial release by 4.4% (0.114 standard deviations). Heterogeneous treatment effect analyses suggest that the program increased strategic litigation among the most experienced public defenders and has distinct effects for those accused of minor and severe crimes. We find no evidence that the mechanism explaining our results involves changes in public defenders’ attitudes or perceptions about their work environments. Criminal justice programs focusing on pretrial detention may help reduce prison overcrowding in high crime countries."

Visual Displays of Uncertainty

A recent post on Gelman's Stats blog (by Jessica Hullman) nicely merges a few critical threads. One involves Tufte's classic push to display data in creative and informative ways (see, e.g., Tufte's classic). Another relates to a detectable nudge towards emphasizing results' "uncertainty" rather than a "false" sense of "precision" through, say, point estimates. While Jessica's discussion dwells on election forecast uncertainty--and interesting ways to visually convey it--the basic point includes obvious spill-over implications. As well, even if one is persuaded that visually conveying uncertainty is a worthwhile objective, the question about whether withholding point estimates a good idea is not easily answered.

Count data, Logging Zeros, and Poisson

A persistent issue confounding many empirical legal studies using count data involves the "zeros problem," especially when the count data require a log transformation. Contests on how to best deal with this problem persist. Popular approaches include standard OLS regression models that add a "plus one" to each count data so to not lose all cases with a zero count in the log transformation and Poisson regressions models.

While empirical assessments of these two approaches are relatively rare, a recent paper compares these two approaches drawing on replication analyses of five recent corporate patenting papers. In Count Data in Finance, Jonathan Cohn (Texas--Business) et al., find that one OLS regression approach (LOG1PLUS) "using log-transformed outcomes can produce biased and incorrectly signed estimates of economic relationships and provide guidance for future research in finance-related applications involving zero-bounded count data." By comparison, while the Poisson model estimates may "lose efficiency if the model’s conditional mean-variance equality restriction is not satisfied in the data, they remain unbiased and consistent as long as the standard conditional mean independence assumption holds." The paper's abstract follows.

"This paper examines the use of count data-based outcome variables such as corporate patents in empirical corporate finance research. We demonstrate that the common practice of regressing the log of one plus the count on covariates (“LOG1PLUS” regression) produces biased and inconsistent estimates of objects of interest and lacks meaningful interpretation. Poisson regressions have simple interpretations and produce unbiased and consistent estimates under standard exogeneity assumptions, though they lose efficiency if the count data is overdispersed. Replicating several recent papers on corporate patenting, we find that LOG1PLUS and Poisson regressions frequently produce meaningfully different estimates and that bias in LOG1PLUS regressions is likely large."

Diff-in-Diff

Owing to the steadily increasing popularity of Diff-in-Diff models across an array of fields, including law, Stata's newest version (v.17) builds out its Diff-in-Diff modeling capacities (a brief summary tutorial video is here). While Stata's brief video provides a quick introduction, Brady Neal's more detailed Diff-in-Diff modeling video (here) might assist those seeking more comprehensive information. Obviously, Diff-in-Diff models are not without important limitations, including the all-important parallel trends assumption. That such modeling limitations exist, however, do not distinguish Diff-in-Diff specifications from other competing empirical strategies.

More Stata Plotting Options

The newly-released subsetplot command produces an array of scatter or other twoway plots for sub-groups of data. As Nick Cox observes (here): "We all know that if you want to compare relationships graphically between groups of observations, we can superimpose different groups in a single plot, or juxtapose different groups in several plots. This [the subsetplot command] is a hybrid approach combining elements of those two strategies."

To illustrate, consider the following plot of auto weight by MPG:The subsetplot command plots these same data across a selected sub-grouping's outcomes (below, by auto repair record groupings (1-5)):For those interested, to download the [Stata] command type:

ssc install subsetplot

"Count" Variables & Dispersion

In many ELS contexts, count data are noted over-dispersion. In many legal settings this issue presents in the form of a "zeros-problem." When such a "skewed, count" dependent variable arises, various approaches include: negative binomial, Poisson, and zero-inflated regression estimations. A recent discussion of this general topic on the StataList (here) includes a robust back-and-forth about the relative strengths and weaknesses of various approaches. The claim that Poisson regression is not inappropriate when there is overdispersion in the dependent variable is forcefully defended. (For an earlier discussion of this general topic click here.)

A New and Interesting Take on Measuring Case Complexity

Most judicial decisionmaking papers can be criticized for inadequately controlling for variation in "case complexity." Lingering disputes over what "complexity" even means in terms of litigated cases only increase the degree of difficulty in constructing such a control. And even if a consensus on what "case complexity" means existed, constructing a reliable, consistent, and stable measure introduces additional challenges.

Difficulties, conceptual and mechanical, aside, a recent paper, Measuring Supreme Court Case Complexity, makes a helpful contribution. In it, Greg Goelzhauser (Utah State--Poli Sci) et al., both critique an array of earlier efforts and advance their own Bayesian measurement model to construct a latent (Supreme Court) case complexity measure that exploits issue and provision counts from the cases' merits briefs. By measuring case complexity from pre-decision data (merits briefs) the paper argues that its ex poste approach "diminishes endogeneity and post-treatment bias concerns when studying merits stage outcomes." To develop its measure, the authors draw from issue and provision (hand-coded) counts extracted from merits briefs filed in the 1954 through 2017 Court Terms, incident to Supreme Court Rules 24(1)(a) and 24(1)(f).

While perhaps limited to Supreme Court cases, new approaches to a difficult challenge can make important methodological contributions. The paper's abstract follows.

"Case complexity is central to the study of judicial politics. The dominant measures of Supreme Court case complexity use information on legal issues and provisions observed post-decision. As a result, scholars using these measures to study merits stage outcomes such as bargaining, voting, separate opinion production, and opinion content introduce post-treatment bias and exacerbate endogeneity concerns. Furthermore, existing issue measures are not valid proxies for complexity. Leveraging information on issues and provisions extracted from merits briefs, we develop a new latent measure of Supreme Court case complexity. This measure maps with the prevailing understanding of the underlying concept while mitigating inferential threats that hamper empirical evaluations. Our brief-based measurement strategy is generalizable to other contexts where it is important to generate exogenous and pre-treatment indicators for use in explaining merits decisions."

Multi-Level (Mixed) Modeling v. Regress & Cluster

When it comes to decisions between, e.g., a "-mixed- or a -regress and cluster- approach," what is increasingly clear (to me, anyway--and reflected in a helpful StataList discussion here) is that no firm, hard rules exists. What is clear, however, is that model specification decisions need to pivot on a granular understanding of the data and their underlying structure.

One classic context that frequently gives rise to such a decision involves efforts to assess student-level educational outcomes given hypothesized varied influences at the classroom- and school-levels into which the individual students nest. While a regression and clustering specification may yield results similar to a multi-level specification under certain circumstances (e.g., "if intraclass correlations at the school and class levels are very close to zero"), as the comments note: "The -regress- approach, even with -vce(cluster ...)- does not adjust for potential confounding due to systematic differences among classes or schools. The -mixed- model does so."

Panel Data Sets & Serial Correlation and Heteroskedasticity Challenges

For those working with a panel data set noted for its "large N, small T," a recent discussion here (from the StataList) might help save some time when it comes to working through challenges posed by serialcorrelation and heteroskedasticity. While these issues are especially common in corp finance and securities papers, as the latter comments in the discussion imply the magnitude of the problem (for 'short panel' data sets) remains somewhat of a contested point.

Empirical Conjectures About the Effects of Raising the Jurisdictional Amount

As my Civ Pro colleagues periodically remind me, we're likely overdue for an increase to jurisdictional thresholds. Even if they're correct, however, what might be the expected (and unexpected) consequences of various increases to jurisdictional thresholds? A recent paper by Steven Gensler (Oklahoma) and Roger Michalski (Oklahoma), The Million Dollar Diversity Docket, puts these counter-factual questions to data. And what they find should interest more than procedeuralists.

The authors hand-coded data from five (non-randomly selected) federal districts (Minnesota, Georgia Northern, New Jersey, Texas Northern, and Utah). From each of these districts they selected a random draw of diversity jurisdiction cases based on Federal Judicial Center (“FJC”) data on cases completed in 2018, resulting in 2,896 usable cases. The paper's methodological move involves substituting the temporal component in survival analysis with a measure of jurisdictional amount. That is, instead of asking on which day a machine fails, we ask at which jurisdictional amount a case fails. To do so, the authors estimated modified KaplanMeier survival functions and Cox Proportional Hazard Regression models.

The paper's main takeaway is that jurisdictional amounts "are not neutral throttles. They do not modulate all cases evenhandedly. Instead, different areas of law, different parts of the country, and different litigants are more affected by changes in the jurisdictional amount than others." The abstract follows.

"What would happen if Congress raised the jurisdictional amount in the diversity jurisdiction statute? Given that it has been almost 25 years since the last increase, we are probably overdue for another one. But to what amount? And with what effect? What would happen if Congress raised the jurisdictional amount from the current $75,000 to $250,000 or, say, $1 million?

Using modified Kaplan-Meier survival functions and Cox Proportional Hazard Regression models to analyze a novel hand-coded data set of pleadings in 2900 cases, we show that the jurisdictional amount is not a neutral throttle. It does not modulate all cases evenhandedly. Instead, different areas of law, different parts of the country, and different litigants are more affected by changes in the jurisdictional amount than others. As such, setting the amount in controversy at any level entails normative choices about which cases to bless and curse with a federal forum.

Our findings implicate not only policy discussions about diversity jurisdiction in general, but also debates currently underway in Congress to utilize modifications of the jurisdictional amount requirement to channel COVID-19 related litigation into federal courts. We hope our empirical work will inform these debates and enable a new wave of scholarship on the basic functions and functioning of the federal diversity docket. We outline some beginnings of this type of work by exploring radically different avenues of thought that develop and re-imagine the diversity docket anew. They range from incremental inflation-adjustments, to carve-outs, experiments with lotteries, and the radical replacement of an amount in controversy minimum with a maximum."

Regression Discontinuity "Agnostics"

One of Andrew Gelman's persistent "rants" (e.g., here and, more recently, here) involves his skepticism about regression discontinuity. Regardless of how one lands on Gelman's critiques, regression discontinuity designs remain a staple in many graduate economics programs and, regardless, increasingly appear in empirical journals that straddle a range of academic fields. As well, as various comments in the conversations that Gelman's posts ignited make clear, reasonable minds continue to differ a bit on regression discontinuity's efficacy (and proper use).

Nested in Gelman's more recent post on this issue, however, is a much more important, far-ranging, and, frankly, less contested point. Gelman notes: "To put it another way: If you want to make a big claim and convince me that you have evidence for it, I need that trail of breadcrumbs connecting data, model, and theory." Say what you will about the enduring contest over regression discontinuity and its potential to fuel overconfidence with a paper's causal identification strategy, Gelman's larger (background) point that researchers should avoid "making a strong claim that doesn’t make sense, is not supported by the data, and is unlikely to replicate" is an obvious one that warrants attention.

A Quick "Margins" and "Marginsplot" Primer

Editors and referees (including, though with far less frequency, student law review editors) increasingly ask for more intuitive (and, frankly, visual) displays of empirical results. What presently dominates in the Stata world include the margins and marginsplot commands. And for anyone interested in seeing a few examples that illustrate how these two commands can, in Chuck Huber's words, "revolutionize" the presentation of regression (or other) results, I haven't come across anything more helpful than this quick video.

Comparing Coefficients in Different Regression Models

Prompted by a question (here) on Statalist relating to efforts to compare (with a TTest) whether coefficients in two separate regression models systematically differ I stumbled upon the suest command. With the suest command, one can, e.g., regress one model, store its results, regress a second model, store its results, and then compare them with the test command. "This will yield a chi2 statistic with 1 degree of freedom. You can run this for each variable." This will help one assess whether the coefficients are the same in each regression model or not. For a very helpful (and brief) video tutorial, click here.

Testing a "Ratio Problem" in Finance Scholarship

On the heels of a recent post (here) describing a Stata regression command tailored to the unique needs of a dependent variable that takes the form of a ratio/percentage/fraction, comes a recent paper that explores potential problems with finance papers that model ratios. Potentially compounding the problem is that ratios as dependents variables are certainly not uncommon in the finance literature (see, e.g., papers featuring Tobin's Q). In The Ratio Problem, Robert Bartlett (Berkeley) and Frank Partnoy (Berkeley) frame the ratio "problem" in terms of two challenges–omitted variable and measurement error bias–"that arise anytime a researcher uses linear regression to estimate a production function that has a ratio as an output." As the paper notes, "In theory, the [ratio] denominator is not necessarily problematic; in practice, however, there are statistical concerns whenever the output of a production function is a ratio." The paper's abstract follows.

“We use the term ‘ratio problem’ to describe the omitted variable and measurement error bias that can arise when a ratio is the dependent variable in an economic model. First, we show how bias can arise from the omission of two classes of variables based on a ratio’s denominator. As an example, we demonstrate that the widely-cited ‘inverse U’ relationship between managerial ownership and Tobin’s Q is reversed when these variables are included. Second, we show how measurement error in the ratio’s denominator can produce bias. We provide empirical tests and solutions, and urge caution about ratios as dependent variables.”