Empirical Legal Studies

Adjusted Predictions and Marginal Effects Resource

On the Stata listserv I recently stumbled across a resource (here) for anyone looking for a helpful "how-to" explanation of Stata's margins command and the resulting adjusted predictions and marginal effects output. Credit goes to Richard Williams, a sociologist at Notre Dame for sharing his slides.

Getting Underneath Data and Results

Over the years I have repeatedly emphasized to my students in empirical methods classes of the need to "get underneath" the data and results, particularly for secondary analyses. By that I mean researchers invariably benefit from on-the-ground insights into and outside perspectives on what their data (and research design) actually capture and results suggest. Kyle Graham's helpful post over at Concurring Opinion illustrates how this general point can work when evaluating a possible empirical project.

An Argument For Using Poisson (With Examples)

While wanting to avoid igniting something of a statistical holy war, I did want to point to an argument from the good folks at Stata for using poisson over log-linear regression  (and negative binomial regression) in some cases. The post (here) quite helpfully includes suggested code as well as Stata output illustrations.

Competing Takes on ELS and "Complexity"

We previously posted Dave Hoffman's (Temple) thoughts on potential problems flowing from an increasingly technical turn in much of the ELS literatures. Corey Yung (John Marshall) pushes back a bit here.

While Yung "completely agree(s) with his (Hoffman's) conclusion that empirical legal studies should seek to be more accessible (which I always note at the end of my introduction of my empirical work), I disagree with his contention that empirical legal studies are facing widespread incomprehensibility due to growing complexity." In fact, according to Yung, in some areas (e.g., judicial decisionmaking) current empirical work has "barely scratched the surface." That is, such work needs to become more technical to get at the dynamics with greater accuracy. Anyway, despite both posts nested in the admittedly trendy "Moneyball" and sabremetrics contexts, they engage with an interesting set of issues.

Another Reason Why Selection Effects Are Important

In his post over at Marginal Revolution Alex Tabarrok reminds us all why selection effects are important to consider in our work.

During WWII, statistician Abraham Wald was asked to help the British decide where to add armor to their bombers. After analyzing the records, he recommended adding more armor to the places where there was no damage.

The RAF was initially confused. Can you explain (without peeking below)?

Because Wald had data only on the planes that returned to Britain he concluded that the bullet holes that he saw were all in places where a plane could be hit and still survive. Wald then surmised that the planes that were shot down were probably hit in different places than those that returned. Thus, Wald recommended adding armor to the places where the surviving planes were lucky enough not to have been hit.

Are the Futures of Empirical Legal Studies and "Moneyball" (Sabremetrics) Joined at the Hip?

Dave Hoffman (Temple) has an interesting post at Concurring Opinions that considers whether ELS and sabremetrics are destined to suffer from similar futures. Dave worries that: "sabermetricians are devoting oodles of time to ever-more-complex formulae which add only a small amount of predictive power, but which make the discipline more remote from lay understanding, and thus less practically useful.   Basically: the jargonification of a field." Although I hope Dave's worry is misplaced I understand his point. That said, perhaps awareness of a challenge ex ante can help reduce the probability of it presenting.

More Musings on "What is Empirical?"

Although I tend to steer clear of such debates, over at Concurring Opinions David Fagundes (Southwestern) discusses the issue. Semantics aside, partly driving David's interesting post is his concern that:

“'Empirical' is not just a neutral term that happens to describe a particular methodology.  It may be understood to connote, rightly or not, a certain degree of rigorousness and exactitude that can set it apart from, and perhaps even above, other methodologies."

While I understand David's point, I'm not sure I buy the premise underneath it. My own view (admittedly, only a datum) is that good scholarship is good scholarship, regardless of the methodological approach. And I am reasonably confident that my take on this is shared by others.

Contested Views of "Tw-Iqbal"

Months ago, the Federal Judicial Center (FJC) released its well-publicized study of Twombly-Iqbal's (or "Tw-Iqbal" for civ pro aficionados) impact on motions practice.  The FJC study concluded that, on balance, the Supreme Court’s new pleadings rules changed relatively little. Responding to the FJC study, Lonny Hoffman's (Houston) paper questions the FJC paper's methods and conclusions. An excerpted abstract (of the Hoffman paper) follows.

"... Three primary assessments are made of the FJC’s study. First, there are reasons to be concerned that the study may be providing an incomplete picture of actual Rule 12(b)(6) activity. Even if the failure to capture all relevant motion activity was a non-biased error, the inclusiveness problem is consequential. Because the study was designed to compare over time the filing and grant rate of Rule 12(b)(6) motions, the size of the effect of the Court’s cases turns on the amount of activity found. Second, even if concerns are set aside that the collected data may be incomplete, it misreads of the FJC’s findings to conclude that the Court’s decisions are having no effect on dismissal practice. The FJC found that after Iqbal, a plaintiff is twice as likely to face a motion to dismiss. This sizeable increase in rate of Rule 12(b)(6) motion activity represents a marked departure from the steady filing rate observed over the last several decades and means, among other consequences, added costs for plaintiffs who have to defend more frequently against these motions. The data regarding orders resolving dismissal motions even more dramatically shows the consequential impacts of the Court’s cases. There were more orders granting dismissal with and without leave to amend, and for every case category examined. Moreover, the data show that after Iqbal it was much more likely that a motion to dismiss would be granted with leave to amend (as compared to being denied) both overall and in the three largest case categories examined (Civil Rights, Financial Instruments and Other). Employment Discrimination, Contract and Torts all show a trend of increasing grant rates. In sum, in every case type studied there was a higher likelihood after Iqbal that a motion to dismiss would be granted. Third, because of inherent limitations in doing empirical work of this nature, the cases may be having effects that the FJC researchers were unable to detect. Comparing how many motions were filed and granted pre-Twombly to post-Iqbal cannot tell us whether the Court’s cases are deterring some claims from being brought, whether they have increased dismissals of complaints on factual sufficiency grounds, or how many meritorious cases have been dismissed as a result of the Court’s stricter pleading filter. Ultimately, perhaps the most important lesson to take away from this last assessment of the FJC’s report is that empirical study cannot resolve all of the policy questions that Twombly and Iqbal raise."

Sociology, Social Science & The Wal-Mart Litigation

Whether the American Sociological Association (ASA) "missed an opportunity to advance sound scientific methods and opinions in litigation when it failed to scrutinize the content of, and method behind, Dr. Bielby’s case-specific claims in the Wal-Mart case" is a contested proposition. Rather than pick a side, I instead urge interested readers to assess the two recent and competing papers. Gregory Mitchell (UVa) et al.'s paper criticizing the ASA is found here; Laura Beth Nielson (Northwestern) et al.'s rebuttal paper here. Comments welcome.

Helpful, Practical Stata Tips

While I take no side (or sides) in the proverbial "Stata v. SPSS v. R" wars (I use all three), I want to note Stata's recent venture into the Blog universe. Though it's far too early to see if the Stata blog will succeed, it's an interesting development nonetheless. A recent post provides a helpful step-by-step guide to one of the more common (yet error-prone) data tasks: merging data sets. The explanation is helpful and includes all too common pitfalls.