Empirical Legal Studies

Thanks for the welcome

I've been a regular reader of the blog since its inception.  I don't know how much actual posting I can contribute -- I have found some of the greatest value from the blog comes from our guest commentators, such as the current discussion of statistical significance. 

While I have for some time agreed with the commentators' contentions, I would add one perspective from an institutional/legal approach.  Any decision rule, be it a law or regulation or standard for evaluating research is going to be inaccurate.  It can't take into account the diversity of circumstances to which the rule would be applied.  And the reason for having such a rule is that it will still be more accurate than the free-for-all of unrestricted case-by-case evaluation.  While the statistical significance rule is substantially flawed, what would be better?  Professor Henkel has provided valuable explanations of how the standard is misused inferentially, but I wonder if there is not a better standard (rule) that could be deployed for research.  I think the overemphasis on statistical significance will continue until a better rule is designed.

Welcome to New Permanent ELS Blog Editor: Frank Cross

On behalf of the ELS Blog Editors, it gives me great pleasure to welcome Frank Cross as a new permanent Editor.   As many ELS folks already know, Frank is a leading empirical legal scholar who holds appointments in Texas' law and business schools.  Frank's scholarship traverses several fields, including the economics of law and litigation, administrative law, and environmental law. His descriptive and normative studies of judicial decision-making have received particular attention for their important insights, creativity, and methodological rigor.  A brief snippet of Frank's representative publications illustrates his scholarly breadth and productivity.  Since 1998, he has published more than twenty articles, including "What's Not to Like (About Being a Lawyer)?" (Yale Law Journal, 2000) (with C. Silver), "Institutions and Enforcement of the Bill of Rights" (Cornell Law Review, 2000), "Realism About Federalism" (New York University Law Review, 1999), "A Modest Proposal for Improving American Justice" (Columbia Law Review, 1999) (with E. Tiller), "Shattering the Fragile Case for Judicial Review of Administrative Rulemaking" (Virginia Law Review, 1999), and "Judicial Partisanship and Obedience to Legal Doctrine" (Yale Law Journal, 1998) (with E. Tiller).  As well, the ELS Blog Editors noted that Frank's contributions to this blog (and others) have been unfailingly constructive, collegial, and probative.

As the ELS Blog approaches its 100,000 site visitor, the Editors felt it an appropriate moment to improve what we hope readers and participants conclude is a helpful and worthwhile endeavor. The addition of Frank as an Editor accomplishes just that.

The social science controversy over significance testing

The social science controversy over significance testing arises from two, often overlapping, sources so there is arbitrariness in saying a criticism of significance testing stems from either source. One source concerns the technical requirements of significance tests, the second source is essentially a philosophy of science question, whether significance tests provide information of value in developing and validating scientific theories (call this process scientific inference). There are those who feel that in social science research, one rarely meets the technical requirements for appropriate use of significance tests and thus such tests should not be used and of course there are those who hold that significance tests can legitimately be used. There are those who feel that significance testing in social science, even if technical requirements are met, contributes nothing of value to the process of scientific inference and of course there are those that hold that significance testing is a necessary and important aspect of scientific inference.

The Uses of Significance Testing

Professor Henkel has set out the basic logic of significance testing. His background, however, includes much thought and experience connected with the uses and misuses of testing, a topic to which I hope he will return in this discussion. Let me start by offering a few thoughts of my own on this slightly more advanced topic.

As I see it, there are two legitimate technical uses of significance testing in social science, inference to a population parameter and inference to causality. The first is the one that is chiefly covered in Prof. Henkel's introduction. Some would say that the second use, causality, is just a subset of the first one. Be that as it may, I think it's at least convenient to view it separately.

To use testing in the first way, one must have a probability sample, call it a random sample for convenience, of a specific larger population. One calculates a statistic on the sample and uses testing to make an inference to the population. In the overwhelming majority of cases in social science, this inference is about a relationship between two or more variables. Again in the overwhelming majority of cases, the null hypothesis tested is that the relationship in the population is zero. If the result is significant at, say, the 5% level, then we can infer with small chance of error that the relationship in the population is not zero. This use is completely legitimate. It comes up in survey research based on random samples all the time. Note, however, that the test tells us nothing about whether the nonzero relationship in the population is causal. Most survey researchers know that they have to jump through a lot of hoops in order to reach this further conclusion even tentatively.

Social Science Statistics Blog

Last week I discovered (for myself anyway) the Social Science Statistics Blog, which is run by a group of hot-shot graduate students from various departments at Harvard, including Health Policy, Government, Political Science, Sociology, and Statistics.  Most of them appear to have an affiliation with the Institute for Quantitative Social Science.  Their posts have a humor, edginess, and candor that is hard not to like--will they be this interesting when they all get academic jobs?  I hope so.

Here is the introduction to an interesting post by Jens  Hainmueller  (government), entitled The Role of Sample Size and Unobserved Heterogeneity in Causal Inference:

Here is a question for you: Imagine you are asked to conduct an observational study to estimate the effect of wearing a helmet on the risk of death in motorcycle crashes. You have to choose one of two different data-sets for this study: Either a large, rather heterogeneous sample of crashes (these happened on different roads, at different speeds, etc.) or a smaller, more homogeneous sample of crashes (let's say they all occurred on the same road). Your goal is to unearth a trustworthy estimate of the treatment effect that is as close as possible to the `truth', i.e. the effect estimate obtained from an (unethical) experimental study on the same subject. Which sample do you prefer?

See here for the rest of Jens' post.

A Brief Overview of Statistical Significance Testing

My training is in sociology and, though I have an undergraduate background in mathematics and a graduate background in statistics, I do not consider myself a statistician. As a result, my perspective on significance testing may be different from that of the professional statistician and those trained in the physical sciences or business.

To better understand significance tests and their use and misuse, it seems useful to provide a relatively elementary level presentation of both where significance tests fit in the framework of statistics and where statistical inference fits in the framework of scientific inference as well as other contexts in which statistical inference is used. Accomplishing this elementary (oversimplified) presentation involves a consideration of what comprises statistics, some statistical terminology and some criteria that determine if significance tests are properly used, along with a little philosophy of science to set the stage for evaluating the utility of significance testing in, at least, the social sciences. The proper use of significance tests minimally involves discussions of sampling procedures while considerations of the utility of significance testing involves understanding the distinction between statistical and scientific inference.

Blog Forum – Significance Testing

For scholars interested in empirical research, no number is more important than .05. To paraphrase a 1967 article, the difference between .05 and .06 is the difference between right and wrong, creditable and embarrassing, success and failure. See James K. Skipper, Jr., Anthony L. Guenther, and Gilbert Nass, The Sacredness of .05: A Note Concerning the Uses of Statistical Levels of Significance in Social Science, 2 American Sociologist 16-18 (1967). When something is significant at the .05 level, we know it must be important.

Well, maybe this is not quite right. So this week two guests will share their thoughts on significance testing and how best to think about it. First, we have Ramon Henkel of the Department of Sociology at the University of Maryland. Along with Denton E. Morrison, Professor Henkel co-edited The Significance Test Controversy (Aldine 1970), which was just reissued last year. (It includes the previously mentioned Skipper et al. article.) Second, we have Lawrence Mohr of the Department of Political Science at the University of Michigan. Professor Mohr is the author of Understanding Significance Testing (Sage 1990).

A big ELS Blog welcome to them both.

C.J. Robert's "Constitutional Crisis" and the K Street Problem

On page 1 of his 2006 Year-End Report on the Federal Judiciary [hat tip to WSJ Law Blog], Chief Justice John Roberts claims that lagging pay ($165K for a District Court Judge, $175K for a Circuit Judge, $203K for an Associate Justice, $212K for the Chief Justice, see here) has "reached the level of a constitutional crisis that threatens to undermine the strength and independence of the federal judiciary."  (In the 2005 Report, judicial pay was merely a "direct threat to judicial independence.") The Chief Justice's statement is troublesome on its face.  But it is also empirically problematic.

First the text.  Presumably, the "strength" of the judiciary would be undermined by reducing the number of prospective judges.   Most law professors make less than $165,000 per year, as do virtually all civil servants and a large proportion of lawyers in private practice.  This is a pretty deep talent pool.  Why isn't it deep enough?  The Chief Justice needs to explain how another $50K or $100K a year will attract qualitatively better judges and thus avert the constitutional crisis.

What about "independence"?  Are judges who endured an FBI background investigation and congressional hearings really going to curry favor with advocates or litigants in the hope of cashing out with a big law firm or a Fortune 500 company?  Short of bribery, what else could "undermine ... the independence of the federal judiciary" possibly imply?

In the 2006 Report, the Chief Justice argues that the federal bench is on the brink of becoming a mere a stepping stone to lucrative post-government employment.  However, isn't this a problem that affects virtually every government lawyer who works in Washington, D.C.?  Single family homes in Bethesda, or Falls Church, or Arlington, or Georgetown, are prohibitively expensive on a government pay scale; so is private college tuition (if you want to pay cash).  As a result, lawyers at the DOJ, FTC, SEC, EPA, and Capital Hill, etc.,  now routinely parlay their government service into lucrative law firm and K Street lobbying jobs. 

So, if a constitutional crisis is afoot, isn't the cancer even more severe in the executive and legislative branches?  The Chief Justice has lifetime tenure--if this is the true dynamic, he should speak up. 

My empirical analysis, and some interesting graphics, after the jump.

Many Thanks

As my time as guest blogger comes to a close I would like to take some time to thank all of those who  read my sometimes too long posts and I'd especially like to thank those who chimed in with comments. I very much appreciate the opportunity provided to me by the ELS editors to correspond with all of you in this blog forum.  Have a nice weekend! -- Jeff

Josh Wright named FTC Scholar-in-Residence

Joshua Wright, an economist and law professor at George Mason University School of Law (and Truth on the Market blogger), was recently named Scholar-in-Residence at the Federal Trade Commission.   (News release here.)  Josh will be at the FTC full-time next year.  This is a great opportunity for an academic to steep himself in real-world policy issues and better understand the institutional players involved.  In turn, the FTC will benefit from Josh's theoretical and empirical skills  and his numerous connections within the academic community. 

These are valuable synergies, and I hope it is the beginning of a trend.   Congratulations and good luck to Josh.