Empirical Legal Studies

Assessing Normal Distributions

A nice illustrative discussion (here) on various approaches to assess whether "normal" distribution assumptions plausibly hold. Participants in the discussion provide a helpful illustration (below) on two perspectives (kernel density vs. qnorm) on a possible bi-modal distribution.

Graphing Counterfactual Regression Trend Lines

In a recent Statlist discussion (here), the question was posed about how to: "generate a graph to show how to the GDP per capita trend line would have continued if the conflict [or any discrete event] had not occurred." (emphasis added) Put slightly differently, the question involves how to over-lay two separate time-trend lines: one from actual data reflecting the influence of a discrete event and another trend line reflecting a counterfactual regression trend (that is, as if the discrete event did not happen).

To illustrate, the graph (above) plots and compares "the effect that the civil conflict in Mozambique and Rwanda have had on the GDP per capita growth in the long run" and over-lays a projected regression line reflecting GDP per capita assuming that the civil conflict did not emerge in these two countries.

While the Stata graphics coding can be tricky, the end result is a quickly interpretable graph.

How to Deal with an Ordinal Dep Var When You Need/Want to Include Fixed Effects?

A recent question posed on StataList has stimulated an interesting discussion (here) as well as a call for FE ordinal models. The specific question:

"So to sum up my question: Is it possible to 'argue' that my ordinal dependent variables can be treated as continuous and I can therefore use FE for them as well? If yes - could you point me towards the appropriate literature/methodology? Or is this not possible and I need to use logit/probit for the ordinal dependent variables?"

The core of the question--can an ordinal dependent variable be treated as "continuous"--comes up most frequently in the survey context where a response involves a Likert Scale (typically a 5- or 7-point scale -- or an even-numbered scale that pushes respondents off a mid-point response). While reasonable minds can--and do--differ on what modeling options may plausibly exist, at the very least, and as a first step, a researcher would need to closely examine how any ordinal dependent variable distributes as this may itself limit model selection options.

Souter's Pen

I guess it's not totally irrelevant that Chris posted about Souter, given the discussion of the ABA ratings and the new vacancy on the Supreme Court he just created...though the pen thing would not, I think, be predictive of either decision making or nomination...  Anyway, I wanted to take the opportunity the pen post created both to thank Amy and Rich for spending some time with us discussing their research, and to think about the ramifications of that research for some of the names being batted around for Souter's seat.  I think we all agree that the pick will be a female, which means, according to Rich and Amy's research, a lower expected ABA rating.  In addition, the pick may well be a non-judge (Jennifer Granholm, anyone?), which would also lead to a lower ABA rating.  However, the pick is being made by a Democrat...So, Rich and Amy, what does your research say about the rating a Granholm might receive?  Does ideology trump judicial experience?  [Perhaps tangentially, will the President letting the ABA back in on the process make their determination more relevant to the Senate?  Does the Senate distinguish between WQ and Q?  Or does it just not vote for someone who is not qualified?]

Discussing "Bias and the Bar" II

Sara - We definitely need to think more about how we can operationalize some of the features of "judicial temperament." I personally need more time to really think this through, so I instead thought it might be good to write about the question that keeps running through my head as I try to operationalize judicial temperament. The issue I'm struggling with, and which I'd love to hear the thoughts from others, is how we guard against imputing too much to the obvious proxies we'd have to use. For example, to play devil's advocate, I'm interpreting your suggestion of using the Congressional staff variable to reflect the quality of "open-mindedness" as implying that the political nature of the job might actually point towards close-mindedness. The question is whether that is reading too much into what it means - or what the ABA perceives it to mean - when someone has served on a Congressional staff. We thought about it in two terms: (1) an obviously political position, but also (2) a not-always-good substitution for traditional legal experience, especially given the political nature of the job. Put another way, did the person get the nomination because of their political service, or due to their qualifications? However, is the acknowledgement of the political flavor of the appointment and the nominee's background the same as measuring whether a nominee will be viewed by the ABA as "open-minded"? Alternatively, by this definition, our inclusion of the Years as Public Interest Lawyer could be seen as another measure of potential "open-mindedness" or "commitment to equal justice" given the equal justice-nature of much public interest work. But, again, are we presuming too much to suggest that the ABA views such work in this manner? Finally, what are service as a government attorney? The vast majority of people in this category - which we found the ABA viewed positively - serve as some type of prosecutor. Is this a signal of substantial courtroom and/or litigation experience, or could one possibly somehow link such experience - either positively or negatively - to some of the suggested ways of understanding "judicial temperament" in the ABA's guidelines? More broadly, how do we make sure that even our operationalization of these terms does not in and of itself make judgments? For example, does it (or should it) matter who the staffer worked for in Congress? Does it matter what issues the staffer focused on?

Of course, much of this relates back to Sara's initial post about the impact of the ABA's investigations - it's not merely that someone was a judge, but that the ABA attempts to determine whether they really were a fair and impartial judge. The problem, however, is that we have little to work with on the outside. We could obviously attempt to look into the information brought forth by others (i.e., the media, interest groups, senators, etc.), but (a) most nominees receive little attention if any, and (b) the information we hear publicly about is likely to be somewhat biased. In fact, as a side note, a big problem is the lack of information on nominees -- when I was constructing a list of nominees who were opposed by interest groups, there were a surprising number who has zero articles written about their nominations, including passing mentions that they were nominated and/or confirmed. Which brings us back to the issue of how do we attempt to explain something - or some idea - for which we only have limited information and potential proxies?

Thoughts?

Finally, we will certainly work on creating a more nuanced ideology measure. The one question, Sara, is were you referring to the distance between the ABA committee and the nominee or the distance between the Judiciary Committee and the nominee? The issue is that, and forgive me if I’ve totally misunderstood your post, is that I'm just not sure what it would mean to hypothesize that the ABA Standing Committee's ratings are influenced by how ideologically distant the nominee is from the Senate Judiciary Committee. This would in turn have to mean that we believe that the ABA is more likely to rate nominees highly in times of unified government (when the ideological distance would be closer), and rate them less well in times of divided government (when the ideological distance would be farther). And, this would suggest the ABA's ratings are in part driven by the makeup of the Judiciary Committee, rather than based on the nominee's background.  Obviously, if we could create some measure of the ABA Committee's ideology that we could then compare to the nominee's ideology, that would be great.... Last thought: yes, determining whether the bias has switched over time is definitely a worthy question; the issue, as always, is getting the data.

Sara - thanks again for really pushing me/us to think about these issues!

Re: Chris's post yesterday, the only possible response is to say thank you for such a great suggestion, and the reminder to pay close attention to how one describes what their model actually does - and for providing us with all of the information on how to put that suggestion into action! We appreciate the guidance!! And, we'll let you know what we found as soon as we're done.

Thoughts on "Bias and the Bar"

Rich, Amy, and Susan have done a great service by writing their study.  My comments here focus on a specific aspect of the paper: the issue of causality.  The paper itself makes a number of explicitly causal claims, referring (for example) to the "most influential causal factors" that might determine ratings, and to the construction of "a series of causal models" (p. 11).

My narrow point is that models of which they speak aren't really causal models at all; they are straightforward regression-type models, of the sort social scientists do a lot of.  That isn't a criticism of what they've done, but rather just an observation, albeit one that suggests a simple change in approach.

Manipulability theories of causation (perhaps most famously set out in Holland's (1986) paper) -- upon which most of the current approaches to causal inference are based -- certainly have their critics.  But they have a natural application to the central question of the paper: What ABA rating would nominee X have received had s/he been appointed by a Democratic president, rather than a Republican (or vice-versa)?

This (to me) suggests that the paper is an obvious candidate for a matching-based analysis of the influence of (e.g.) party on ABA ratings.  Such an approach is (a) very easy to do, as a practical matter, (b) allows one to control for other potentially confounding factors, and best of all (c) provides a direct answer to the question posed above that can be interpreted in causal terms.  These approaches are increasingly widely used in empirical legal studies, and seem (to me) to hold particular promise in addressing the authors' central question.

Discussing "Bias and the Bar"

Thanks for your post, Sara! You are obviously correct in that we cannot truly capture in our model the full parameters of the investigation the ABA conducts. Most importantly, we do not know what is learned in the background interviews, and the ABA does not share that information with anyone outside of the ABA committee, including the president and the Senate Judiciary Committee members. However, we would argue that our model does more than simply capture nominee qualifications per se since the crucial question is not how would others rate these nominees, but how did the ABA evaluate them? The issue is that what is considered "qualified" really does vary based on who you ask, and, as evidenced by external discussions of nominee ABA ratings, the ABA's ratings do not merely reflect the conventional wisdom as to whether a certain person is, objectively, qualified for the federal bench. We do examine information others can also find, but we chose these measures because these are the types of qualifications the ABA itself says it considers in its guidelines. Furthermore, even if others possess this same information, the reason the ABA is granted this role is its unique ability to evaluate how much these different background experiences matter. Senators thus rely upon the ABA to give them guidance about whether a particular nominee is, in fact, fit for the federal bench.

The other important issue is that Senators depend on the ABA because they lack the time, resources and staff to do their own investigations. Even if the senators have nothing more than the ABA rating, the belief that the rating reflects an in-depth evaluation heightens the importance of this rating. And, even if our model is merely capturing what factors are likely to lead to a determination that a nominee is qualified per se, since it is the ABA that sets the gold standard for this discussion, then even capturing that part of the process provides important insights.

As to whether the findings are time-bound, our reason for looking at nominees from 1985 onward was to deal with any potential time-based issues. Since 1985, and especially since the Bork nomination in 1987, criticisms directed at the ABA have been from conservatives. We therefore believe our findings explain the process as it exists today, and most likely, the process in the future, but agree that we cannot necessarily use them to explain the ABA's behavior in earlier periods. And, given important changes such as the increased politicization of lower court appointments, the increased politicization of the ABA House of Delegates, and changes in the evaluation standards themselves, we think it best to focus our attention on this more recent period.

Omitted variable bias is always of concern, and certainly our model cannot capture how the additional context provided by the ABA's committee's interviews influences the decision-making process. That said, our model does, we believe, reflect the influence of the major factors the ABA considers, and provides us with insights on how the ABA determines whether a nominee is considered "qualified' for the federal bench. If the factors we examine only point toward whether a nominee is qualified per se, that in fact suggests that we need to understand even more why we then find differences based on ideology and partisanship.

Forum Post #8: A Few Concluding Thoughts

Jason and I haven't had a chance to coordinate on this, so rather than trying to speak for both of us, I'll offer my thoughts and he can chime in with his own if he is so inclined.

First, I'd like to thank everyone for their thoughtful comments and critiques of our paper.  You've given us a lot to think about and the next draft of the paper will be substantially better because of your input.

Second, there's been a lot of talk over the last two days about ways in which our data might be refined and improved.  There is clearly a great deal of useful work that could be done in this area and I hope that our study will just be the beginning of a more well-informed discussion of the student-edited law review and how it fits into the overall schema of legal scholarship.    I, for one, would be particularly interested to seem some empirical data, expanding on Christine's "armchair empiricism," that examines what law reviews actually publish.  Somewhere between what editors say they consider and what they actually publish lies the truth about how these decisions get made.

Finally, I want to thank Bill Henderson for seeking us out and organizing this forum.  It is, of course, always nice to have someone come from out of the blue and express interest in your work.  But the forum has also been useful for us and, I hope, for everyone else as well.

Forum Post #6: Moving From Anecdote to Data

First, thanks to Bill and everyone for letting an empirical neophyte trespass on this blog for awhile.  As a typical junior professor, I am a regular submitter to law reviews and have spent untold hours talking to colleagues about the submission process.  However, lately these discussions have grown tiresome as I realize that as arbitrary and random as the process seems to be, unrelated anecdotes only make the process seem more random instead of creating some pattern of order.  No one person's "N" is ever going to be big enough for that person to say with any authority "The best way to get your article accepted at top journals is to do X."  However, these conversations seem always to dwindle to the point where Professor A says that doing one thing is important, but then Professor B will counter with the argument that the Professor A must be wrong because Professor B never does that one thing and always places well.  And so on.  So, I am pleased as punch to be moving from war stories to data.  If nothing else, the student-run law review system produced the two authors of this study and perhaps honed their writing skills to the point that they were able to advance knowledge in this way.  (I would like to hear about their own placement story, though, now that they are on the other side!)

However, as everyone else seems eager to point out, survey responses aren't the sort of data we need to conclusively dispel or confirm these anecdotes.  (As an alumnus of the Lee Epstein/Andrew Martin Conducting Empirical Legal Scholarship workshops, I remember one of them saying that the best way to introduce bias into your project is to do a survey!)  I won't belabor the point, but I think we all know that law review editors are smart enough to know what answers they should pick.  (However, a survey of former Articles Editors might produce less self-conscious responses.)  I think it's interesting that the "Tier 1" editors assigned lower importance scores to certain categories regarding author prestige and article topic, but those factors were still the most important in rank order.  And, as the authors note, when asked virtually the same question in different form regarding author's place in the world, editors answered differently depending on wording.  They don't care about how "notable" an author is, but they care whether the author is "highly influential in her respective field."

So, we know now that articles editors, even though they understand that they shouldn't say it's super important, admit that characteristics of the author are pretty much as important as characteristics of the work.  So, that seems to jive with everyone's worst fears and conventional wisdom.  And I think the objective data would support that.  A few weeks ago, I updated some armchair empiricism on which authors get published in the Harvard Law Review (not a respondent to the survey).  While my interest was in the gender breakdown of the authors published, I hasten to add that any casual legal scholar would recognize most of the names of those whose work was published in any given volume.  I would suspect the same would be true of at least the top 5 journals.

So, what can I possibly add to this forum?  Here are two small bits.  First, I think the "hot topic" factor here is not treated in a precise way.  Although the authors suggest that at least most journals do not look for hot topics, the factor was phrased as "The topic is one about which many articles are currently being written."  Well, that sentence does not cry out for agreement, and one can think of many other ways in which the same could be worded.  In fact, two factors that were ranked highly ("The article fills a gap in the literature" and "The topic would interest the general public") also describe hot topics.  One can imagine that a legal topic that is currently in the news would spawn articles that both fill a (new) gap and that would interest the general public.  Because of the timeliness, this topic may be the subject of many law review articles, but it's hard for an articles editor to know that at the beginning of the trend.

Second, one interesting aspect about the law review selection process is the agency problem.  Editors are choosing articles that they will work on, perhaps personally.  While the authors assume that these individual editors choose articles that will gain their law reviews citations and attention, in reality these editors will move on before this attention happens.  So, editors may be more interested in choosing topics that they themselves are interested in and possibly choosing authors that they would like to get to know.  I would be interested to know if any articles editors thought about the relationship between themselves and their authors.  I definitely remember the authors that I worked with.  And, I could definitely tell when I was getting an offer from an articles editor who really liked my topic.

Forum Post #4: What is Right About Student-Edited Law Journals?

One of unwritten questions raised by the Nance-Steinberg study is whether the institution of the student-edited law review remains the appropriate outlet for the "best" legal scholarship, including interdisciplinary and empirical work.   

Nance and Steinberg, for example, cite to Judge Posner's remark that the shift away from doctrinal scholarship has left Articles Editors floundering in a "scholarly enterprise vast reaches of which they could barely comprehend."  According to this view, student-edited journals should focus on doctrinal work while the more sophisticated stuff should be reserved for faculty-run journals.

This argument is quite popular in some law professor circles.  But I think it begs a more fundamental question:  What is the purpose of legal scholarship?  Perhaps this is a better way to phrase the question:  What is the purpose of scholarship produced by law faculty?   

A lot of articles published in prestigious faculty-edited law journals are impenetrable, using formal mathematical modeling that only a specialist could understand.  There is no pretense that the journal is engaged in dialogue with a larger legal community.  Under this paradigm, scholarly "success" is a remarkably insular conversation among elite academics.  Why should a law professor be paid to write articles that graduates of his or her law school cannot understand?  What is the justification for the disconnect between the classroom and the faculty-edited law journal?

Although the current student-edited system has drawbacks, it also has some huge advantages:

• There are no faculty gatekeepers who have built their careers around established theories and methodologies;
• It demands that the insights of empiricism and social science be written in a way that is intelligible and educates the reader (yes, footnotes can educate!), which is appropriate since many lawyers eventually become regulators and policymakers;
• Seriously flawed studies will be ignored by the marketplace of ideas, which also happens in the peer-reviewed world (see, e.g, here);
• Unlike many social science journals, law reviews permit (rather, expect) an author to explore the normative implications of their empirical work--this is a source of envy for many economists, sociologists, and political scientists.

That said, there is ample room for outlets like the JELS, which has become an important vehicle for setting standards for (intelligible) empirical legal work, including the advancement of  methods. 

But regardless of publication outlet, we still need to answer the question, "what is the purpose of our scholarship (beyond our own professional advancement)"? If law students cannot evaluate the merits of our articles that bear on the law, perhaps the problem is primarily one of curriculum (and the institutional incentives that perpetuate an antiquated model) rather than publication outlet.   

Forum Post #3: Methodological Matters

My Ph.D. thesis director once advised me that "if it's worth doing, it's worth doing badly."  His point was not to make the perfect the enemy of the good, particularly when conducting truly original research.  So it's important to preface any critique of their work by acknowledging that, whatever flaws their study may have, Nance and Steinberg have done a great service to the legal academy by shedding some empirical light on the question of law review publication.

As is the case with any empirical paper, methodological criticisms of their work are among the easiest to offer.  One could question their decision to use conventional factor analysis with ordinally-measured response variables (particularly when better techniques exist), or their extensive use of tables when figures (such as the one above; click on chart to enlarge) typically do a much better job of conveying complex statistical results.

My biggest concern, however (and one prefaced by Michael's comment to Bill's first Forum post) is the effect of social desirability bias (hereinafter SDB) on the study's findings.  SDB refers to survey respondents' tendency to answer surveys in manners they think are socially (or, here, professionally) desirable or expected of them; it is a well-known and commonly-observed phenomenon in survey research (a recent paper with a list of current references is Streb et al.).  I'd contend that the presence and effect of such bias can explain both their intuitive findings as well as some of the more unexpected ones.

Articles editors (AEs) undoubtedly are interested in growing the prestige of their journal, and in minimizing their editorial workload.  They are also, however, socialized into the law review culture; they understand that law reviews, as forums for scholarly work, should publish the "best" (most original, creative, important, well-reasoned, persuasive) scholarship they can.  As AEs, their professional role is to select such work for publication, and to do so in a way that doesn't systematically disfavor authors or work on the basis of other (putatively irrelevant) criteria.  SDB suggests that AE's survey responses will likely reflect their desire to be seen as conforming to that role.

Consider Nance and Steinberg's rather odd finding that, while "Author Prestige" is among the most influential of their constructs, "Notability of the Author" ranks dead last in the rankings of publication criteria. The phrasing of the authors' 56 "influence" questions is such that none is dispositive; each can influence the publication process without making or breaking a given paper.  In contrast, asking AEs to rank order the seven publication criteria forces a zero-sum choice: for one criterion to be ranked higher, another must be ranked lower.  That, combined with the relatively small number of items to be ranked and the presence of SDB effects, makes it difficult for an AE to place "Notability" high in the rankings.

A similar dynamic might explain the relative weakness of "negative" author traits: while AEs can be forgiven for privileging work by high-prestige authors, it is considered much less acceptable to disadvantage low-prestige ones.  Finally, in our post-Grutter world, it is likely that most AEs almost reflexively responded "no influence" to questions regarding the effects of author race and gender.

But while SDB is a potentially serious problem, it is by no means insurmountable.  A standard way of assessing the presence of SDB is to compare survey responses with actual behavior; as Michael suggested in his earlier comment, the obvious means of doing this would be to analyze data on actual submissions and acceptances.  Barring that, SDB can be reduced in surveys through anonymity; survey respondents who are assured that their answers will be anonymous are typically less affected by SDB than those who can be identified.

Forum Post #2: Some Context from the Authors

This study grows out of our own experiences as Articles Editors of the University of Pennsylvania Law Review.  The Articles Office spent many of our early meetings talking about what criteria the law review ought to be using to select articles.  In the wake of those discussions, we were curious about the degree to which our peer journals would make use of the same criteria and weigh them the same way.  This study started as a way to answer that question.

Our first startling discovery was our fellow editors’ eagerness to talk about these issues.  We sent out an e-mail to every student-edited legal journal for which we could find an e-mail address and asked one or more of the editors responsible for selecting articles at that publication to fill out our survey instrument.  Though it’s impossible to know exactly how many of those e-mail addresses were valid, we believe that our e-mail reached between 375 and 400 journals.  The 191 responses from 163 journals were far more than we expected and gave us a data set that was well-suited to productive analysis.  We chose to focus on factor analysis because we believed that the specific criteria that were amenable to survey questions were properly understood as components of broader selection criteria.  Our results bore that out. 

While there is obvious interest within the academic community in our simply reporting these results, we think our data also highlight some interesting aspects of the law review process that have been little discussed.  The most notable is the degree to which journals act as independent agents rather than as neutral arbiters of quality scholarship.  (We have consciously avoided taking on the question of whether students are capable of filling a role as neutral arbiters, but commentators have frequently argued that they are not.)  The primary measure of journal prestige (and the one we used in our analysis) is frequency of citation.  If the primary goal of editors is to increase the notability and prestige of their own journal – and our results can be read as indicating that it is – the best way to do that is to publish articles that will be read and cited frequently.  While that goal may correlate to some degree with an abstract notion of academic excellence or importance, it also draws on a number of other factors such as author notoriety or prestige and the frequency with which related topics are addressed in legal academic writing.  Thus, it is possible to explain editors’ tendency to gravitate towards articles by well-known authors at prestigious institutions (and our survey confirms that a strong tendency to do this is present across the board) or to articles in certain subject areas (most notably constitutional law) not as a product of their inability to recognize academic excellence but as the result of rational desire to increase the prestige of their own publications.  As Bill mentioned in his introduction, this is a job that student editors can likely do pretty well.

When undertaking this study, we were struck by the relative dearth of empirical work in this area.  Our hope is that, by providing some robust data about what law review editors actually do, we can move the debate about what they should do away from the anecdotal context of particular authors’ horror stories to a broader context based on a deeper understanding of how the process actually works today.

Forum Post #1: Opening the Black Box of Law Review Selection

Periodically law professors convene forums to trash law reviews. The most virulent words are usually heaped upon the student editors who run these journals. For example, in a symposium published by the University of Chicago Law   Review in 1994, student editors were referred to “incompetents”; in a similar forum in the Stanford Law Review in 1995, they were called “dumb and stupid.”

Within the article selection process, more specific criticism includes the student editors’ (perceived) fixation with copious footnotes, excessive literature reviews, trendy topics, or an author’s institutional affiliation or prior publications (as opposed to the one in front of them). These last items are of particular concern for law professors because it stacks the deck against an objective evaluation of their work. And placement has significant collateral effects on pay, promotion, and the lateral market.

Fortunately, the Nance-Steinberg study opens the black box of article selection. In my estimation, it reveals a reasonably fair and objective process in which original, persuasive, and polished articles have at least a fighting chance of getting a good placement. Sure, the author’s letterhead matters. But these results suggest that this bias is actually less true at more elite journals—and that finding itself may be a bit of a mythbuster. Moreover, the study actually has some suggestions that could, at the margins, help with an article’s placement.

To get all of our readers onto the same page, after the jump I will briefly summarize the sample, methodology, and key findings of the Nance-Steinberg study.

Forum on the Law Review Selection Process

Over the next couple of weeks, the law journal submission season will enter full swing. Because there are only a limited number of publication slots at elite law journals—and there is a perception, if not a reality, that a strong placement affects pay, promotion, and lateral offers—this positional competition creates severe angst among law professors that often ends in disappointment.

To help elucidate the inner-workings of this process, the ELS Blog will be hosting a forum today and tomorrow (August 14-15) on a recent empirical study by Jason Nance and Dylan Steinberg, “The Law Review Article Selection Process: Results from a National Study.”  The authors are 2006 graduates of the University of Pennsylvania Law School, where they served as Articles Editors on the University of Pennsylvania Law Review.  Both authors are currently serving as clerks in the federal judiciary.

This forum will also include commentary on the Nance-Steinberg study by Benjamin Barton (Tennessee Law), Christine Hurt (Illinois Law and The Conglomerate), Ahmed Taha (Wake Forest Law), and our own Chris Zorn. I will begin the forum with a short summary of the Nance-Steinberg study.

Blog Forum – Significance Testing

For scholars interested in empirical research, no number is more important than .05. To paraphrase a 1967 article, the difference between .05 and .06 is the difference between right and wrong, creditable and embarrassing, success and failure. See James K. Skipper, Jr., Anthony L. Guenther, and Gilbert Nass, The Sacredness of .05: A Note Concerning the Uses of Statistical Levels of Significance in Social Science, 2 American Sociologist 16-18 (1967). When something is significant at the .05 level, we know it must be important.

Well, maybe this is not quite right. So this week two guests will share their thoughts on significance testing and how best to think about it. First, we have Ramon Henkel of the Department of Sociology at the University of Maryland. Along with Denton E. Morrison, Professor Henkel co-edited The Significance Test Controversy (Aldine 1970), which was just reissued last year. (It includes the previously mentioned Skipper et al. article.) Second, we have Lawrence Mohr of the Department of Political Science at the University of Michigan. Professor Mohr is the author of Understanding Significance Testing (Sage 1990).

A big ELS Blog welcome to them both.

Case Coding by Students or Faculty

This is the last of our series of five installments on content analysis of judicial opinions, which we hope prove to be useful for those who want to undertake or refine this method down the road.

 

A major advantage of content analysis is the ability to delegate much of the grunt work to student coders.  But our experience, and that of some others we’ve heard from, is that students can be dangerously prone to errors or misunderstandings.  Avoiding these requires laborious stages of piloting, training, documentation, and double-coding.  Is this worth the effort?  Fewer than a quarter of the 125 projects we reviewed used student coders primarily; in the rest, the authors appeared to do their own coding.

 

Our view is that student coders should be used when this adds value and not just to save effort.  In other words, sometimes students can do a better job than faculty can on their own.  Here’s why.

 

Realize, first, that the methodologic purpose of content analysis is to bring scientific objectivity to a reading of cases.  If that isn’t your purpose, then why bother?  The method’s scientific rigor arises from the replicability of coding, which should be demonstrated rather than presumed.  This usually requires double-coding to show inter-coder reliability (or consistency) – i.e., that different readers would find the same content you found. But, it can be difficult for non-expert student coders to agree on the more legally-relevant content of cases.  Faculty coding may be essential when analyzing more subtle, latent, or judgmental -- i.e., more interesting -- aspects of cases. 

 

The reliability problem isn’t solved, however, simply by shifting to faculty coding.  Just because you’re an expert, your coding is no more self-validating than is a student’s.  That’s why, from a scientific point of view, coding by 3 students is more rigorous than coding by the single greatest scholar in the world.  To demonstrate replicability, faculty who do their own coding should also have some of it double-coded independently, perhaps by a similarly-expert colleague.  But, when the subject matter allows for student coding, that may be a better way to go, both for reasons of expediency and reliability.

Selection Troubles in Content Analysis

This is the latest installment in a blog forum on Content Analysis, as applied to the content of judicial opinions.

After Mark and I collected 125 examples of content analysis, we evaluated how well scholars used this technique over the years. As you might expect, authors of the studies have paid attention to the selection of judicial opinions for study. But we think authors are worried about the wrong selection problem: it is common to see discussions of selection bias, instead of more troublesome issues of selection replicability.

Many scholars over the years have discussed selection bias. They wonder if the published opinions they reviewed were typical of the entire pool of written judicial opinions; whether the chosen opinions were typical of judicial decisions more generally; whether the chosen opinions were typical of litigated cases; whether the chosen opinions were typical of underlying legal problems in the world. The connection gets weaker at every step, but you saw that one coming, right?

The proper inferences to draw from a collection of cases must account for these gaps. Still, the selection bias problem is actually smaller for judicial opinions than for many other texts to be studied through content analysis -- the number of relevant opinions is often small enough that the author can code the entire relevant universe.

A selection issue that get much less attention, however, is selection replicability. Most often, these content analysis projects describe the general search criteria that the authors used to identify cases, and just stop there. A smaller group specify the sources searched (e.g., West Decennial Digests for older projects, designated LEXIS/Westlaw databases for newer projects). A small number of projects list the actual Westlaw/LEXIS search terms used. A surprising number of the projects simply listed the cases that were included in the study.

All too often, the description of the case selection method was too sketchy to allow another researcher to replicate the same results. (A simple listing of the cases doesn't exactly do the job.) That basic building block of the scientific method -- replicability -- deserves more attention than the now well-known potential biases that flow from studying the world through a collection of judicial opinions.

It's a important shift in thinking that still hasn't percolated down very far into the legal academic ground.

Good and Bad Uses of Content Analysis

We are not proselytizers of content analysis.  It has certain advantages in legal studies, but also substantial limitations, compared with conventional legal analysis.  A full exegesis would take a dissertation (at least), but here are a few summary thoughts.

We think the most successful applications have been those that:

• Debunk conventional wisdom by showing that widely-held beliefs are descriptively wrong about what’s actually in the caselaw.
• Describe the landscape of a body of caselaw, much like a naturalist might explore a new continent or even a familiar patch of woods, by turning over stones and carefully documenting what crawls out.
• Use regression analysis to uncover hidden patterns in complex fact-specific areas of law
• Study particular techniques of opinion-writing, such as syntax, semantics, citations, or reasoning style.

These applications put content analysis to its best use in legal studies because they highlight the empirical weaknesses of traditional legal methods, and they focus on aspects of opinions that are objective and primarily descriptive.

Less successful, or potentially misapplied, uses are those that try to:

• Predict the outcome of future cases as practicing lawyers might do
• Explain definitively what legal or factual elements cause courts to rule one way or the other
• Bring the appearance of scientific rigor to a normative critique of the law (e.g., to show that courts got it wrong/right or made bad/good law)

The primary problem in the first two on this “bad” list is the inherent bias and circularity (or “endogeneity”, as social scientists would say) in how factual elements are reported in opinions.  Our skepticism is not to disbelieve judges when they give us the reasons for their decision.  Instead, we question whether factual and legal elements can be cleanly separated looking only at an appellate decision.  Also, we doubt that the facts reported are reliable guides to real-world facts because of how they are filtered and distorted by the litigation process.

.

Do you agree that the first batch of purposes is each a legitimate use of content analysis?  Would you like to defend any of the second batch of questionable uses?

Non-PhDs and Content Analysis -- Learning Social Science on the Fly

Are ordinary law professors as well-equipped as PhD social scientists to carry out content analysis of judicial opinions? There are reasons to think that they might not be. Content analysis, after all, is a standard technique used in communications, political science, sociology, and other disciplines. It is a common part of the methodological training of social scientists. Legal scholars, meanwhile, often “discover” the techniques of content analysis during research, intuiting many of its essential components on the fly.

-

On the other hand, there are reasons to think that non-PhD legal scholars could do content analysis reasonably well, even without formal training. In some ways, the method resembles the traditional lawyerly job of reading opinions and trying to evaluate them by using consistent criteria.

-

So which is it? Does PhD training matter in the execution of content analysis?

• First, a word about trends over time. About two-thirds of the 125 studies we located from the past half-century include a PhD author.  Surprisingly, this ratio has not increased in recent years, despite changes in the composition of law school faculties.
• Second, the presence of PhD authors does make a difference in citations to methodological literature. Among the PhD-authored studies, three quarters cited to methodology literature, compared to half of the non-PhD-authored studies.
• Third (and this surprised us), the presence of PhD authors did not make much difference in efforts to demonstrate coder reliability. While it was predictable that 71% of the non-PhD studies included no discussion of coder reliability (such as measuring consistency among coders), fully 62% of the PhD studies also included no discussion of coder reliability. The non-PhD studies used student coders 40% of the time, while the PhD studies used student coders 34% of the time.
• Finally, the studies by PhD authors did tend to use statistical techniques more often to analyze the coded results. About two-thirds of the PhD studies used multiple regression analysis, compared to 40% of the non-PhD studies.

How would you expect PhD-authored studies to differ from those conducted by authors with legal training only?

Content Analysis of Judicial Opinions

We (Mark Hall and Ron Wright at Wake Forest) have been asked to host a discussion forum this week on using systematic content analysis to study the legal content of judicial opinions. If any empirical method is especially well-suited to legal studies, surely it is content analysis since this is the established scientific method for reading any type of text.  Most of us are familiar with several good examples that apply this method to law cases.  But, until we started looking systematically and keeping track -- i.e., doing a content analysis of content analyses -- we had no idea how much this was being done, and for how long.

-

Take a wild guess:  How many published projects use systematic coding to study the legal or factual content of judicial opinions?  To keep you reading, the answer is at the end.  (Teaser: It’s more than a couple dozen.)  And, what are the largest such studies, when were they first done, and who has done this the most?  (Hint: Cornell is not in any of these answers.)

-

We don’t include the many fine examples of “docket analysis,” which code only for information obtained from docket sheets or brief abstracts (subject matter, parties, outcome).  We also don’t include the many interesting studies of courts’ citation patterns.  Instead, we’re interested in research projects that code cases for the same material used in traditional interpretive legal methods:  facts, law, or judicial reasoning.

-

(Drum roll) And the answers are:

• There are 125 distinct coding projects (some with many publications)
• The first was published in the 1957 by political scientist Fred Kort
• The rate jumped from <1/yr before1990, to >6/yr since 1990
• The largest are the State Supreme Court Database (Brace & Hall) and the Court of Appeals Database (Songer).  Each has over 20,000 cases.
• The longest is the United States Supreme Court Judicial Database (Harold  Spaeth), which began in the mid-1980s and is still ongoing (9000 opinions)
• Karl Llewellyn (1960) and Richard Posner (1972) were among the early users of this method
• The most frequent content analyst is Willy E. Rice at St. Mary’s (8 projects).

For more detail, see Systematic Content Analysis of Judicial Opinions.

-

Tomorrow, we summarize how much of this work is being done by law professors vs. Ph.D. scientists.  For now, let us know what you think: Are you surprised by how often content analysis is being used to study the legal content of cases?  And, why do you think this method has catapulted since 1990?

Great Moments in the History of Ignorance & Division IV

Again moving forward through time, today’s fourth and final Great Moment comes from the 1960s, a bit of doom and gloom about the state of public law and legal education. Admittedly, it’s not all bad in this period. In the mid-1960s the Harvard Law Review published a symposium with contributions from five political scientists and a response from Lon Fuller. See Joel B. Grossman et al., A Symposium: Social Science Approaches to the Judicial Process, 79 Harv. L. Rev. 1551-1628 (1965-1966). But this effort at bridge-building seems overshadowed by the extent of the divide between students of judicial behavior and students of legal doctrine. Glendon Schubert elaborates in a thorough discussion of the development and decline of public law as a field. He even offers a few suggestions for making public law in law schools “less ghastly.” Glendon Schubert, The Future of Public Law, 34 Geo. Wash. L. Rev. 593, 614 (1965-1966).

Schubert, whom Jeffrey Segal describes as the “founder of the attitudinal model,” provides many gems in this article. See Jeffrey A. Segal, Glendon Schubert: The Judicial Mind, in The Pioneers of Judicial Behavior 78, 78 (Nancy Maveety ed. 2003). Here are three:

To be perfectly blunt, legal research remains today at the primitive level of development, as a science, that generally characterized political science one or two generations ago. By and large, legal method is no more in phase with modern science than it was in the days of Austin, or (better) Aquinas, or (best) Aristotle, all of the talk about “legal science” and computerized methods of retrieving legal data to the contrary notwithstanding.

Schubert, supra, at 600.

Great Moments in the History of Ignorance & Division III

I started with Frank Goodnow’s 1904 presidential address at the first annual meeting of the American Political Science Association. Today’s Great Moment comes from Charles Grove Haines’ 1940 presidential address at the APSA’s 35th annual meeting. Haines was one of Goodnow’s students, and like Goodnow, one of the great public law scholars. See Glendon Schubert, The Future of Public Law, 34 Geo. Wash.L. Rev. 593, 603 (1965-1966). In this excerpt, Haines describes the growing divide between law and political science:

In the early years, when political science was becoming a separate and independent field among the social sciences, public law, a subject at this time accorded little attention in the curricula of law schools, was for many of the members of the new organization one of the major lines of interest. Gradually the scope of work coming within the ambit of the American Political Science Association has been broadened and enriched by the exploration, analysis, and interpretation of other phases of political ideas and phenomena, with the result that political science may truly be regarded as no longer under the bondage of the lawyers. On the other hand, public law has now become one of the important subjects of instruction in law school curricula, and with this trend has come a decline of the importance of this field of instruction in departments of political science. In fact, the pendulum has swung so far in the opposite direction that there are some who maintain that the only deserving place for public law courses in which the case method of instruction is used is in the law schools. In my opinion, such views are incompatible with the best interests of legal instruction both in the law schools and in departments of political science. Public law always has been and continues to be primarily public in its nature, significance, and implications. It belongs, therefore, as much to the field of the political and social sciences as it does to the field of the law, and political and social scientists as well as lawyers may well cooperate in the consideration and evaluation of the vital problems involved in its development.

Charles Grove Haines, The Adaptation of Administrative Law and Procedure to Constitutional Theories and Principles, 34 Amer. Pol. Sci. Rev. 1, 27 (1940) (emphasis added).

Tomorrow’s Final and Really Great Moment: Glendon Schubert describes legal research as “primitive,” explains why political scientists started ignoring their public law colleagues, and calls the study of public law in law schools “ghastly.”

Solum on Interdisciplinary Ignorance

Over at the Legal Theory Blog, Larry Solum has a response to Frank Cross’ post on interdisciplinary ignorance. One of his conclusions: “[T]he legal academy continues to reinforce interdisciplinary ignorance. As a profession, we are failing badly in the training of future legal academics.”

UPDATE: Daniel Solove at Concurring Opinions responds to Solum responding to Cross.

UPDATE II: Mark Graber posts. And Mark Tushnet reacts.

Great Moments in the History of Ignorance & Division II

While teaching at Harvard in the early 1930s, Hugh Langdon Elsbree published a book entitled, Interstate Transmission of Electric Power: A Study in the Conflict of State and Federal Jurisdictions (Harvard 1931). Elsbree later served as the director of the Legislative Reference Service, the precursor to the Congressional Research Service. He died just two years ago, at age 100.

The relevance of Elsbree and his book is that Felix Frankfurter reviewed it for the Harvard Law Review, and in doing so, he offered some thoughts on the great divide. Frankfurter was not terribly enthusiastic about the book, complaining that Elsbree was “too awed by judicial language.” Felix Frankfurter, Book Review, 45 Harv. L. Rev. 762, 763 (1931-1932). Frankfurter was also unhappy with political scientists and economists in general. He argued that they should attend to their own responsibilities rather than those of lawyers. So here is today’s Great Moment in the History of Ignorance and Division, a call for more division:

Economists and political scientists rightly criticize law and lawyers for their formalism. Legal decisions, say our critics, are too often exercises in dialectic, and do not grow out of the soil of life. This charge finds deep sympathy in my breast. But an even greater dissatisfaction that I have with American public law decisions is the snake-like fascination that their sterile dialectic exercises over economists and political scientists. We lawyers are dependent upon these specialists for a critique of the economic, social, and political assumptions which so often underlie what purport to be technical legal doctrines. Yet, more often than not, instead of subjecting such decisions to the test of economic and political fact and wisdom, the economists and political scientists are subdued by the materials of the law and themselves become legal dialecticians.... What we have a right to expect from economists and political scientists is an analysis of what the true governmental problems are, in the light of what actually goes on in the world and wholly apart from the technicalities of American constitutional law. Only after we have such an analysis is it the function of the American lawyer either to find within the existing body of law resources adequate to reconcile law with wisdom, or, if that reconciliation is impossible, to fashion new law. Until the economists and political scientists attend to their special tasks and we lawyers to ours and each has an awareness of the others’ problems, we shall continue to have what Alvin Johnson calls the cross-sterilization of the social disciplines.

Id. at 763-64.

Tomorrow’s Great Moment: Charles Haines, another past president of the APSA, declares that political science is “no longer under the bondage of the lawyers.”

*For more on Elsbree, here are page 1 and page 2 of the tribute offered to him by Representative Robert Ney in the Congressional Record.

Great Moments in the History of Ignorance & Division I

This week, I’ll offer some historical highlights on the current blog forum topic, the divide between law and political science. I’ll rely first on Frank Goodnow (1859-1939) for some thoughts on the relationship between the two disciplines at the turn of the century, a time when there was much less division.

Goodnow, the APSA’s first president, was a member of the Department of Public Law and Jurisprudence at Columbia. Along with John Burgess, Munroe Smith, and John Bassett Moore, he was one of the leading members of the department. Like Goodnow, Smith and Moore served as APSA presidents. And all four held joint appointments in the law school. During this early period of organized political science, public law courses were often of greater interest to political scientists than law professors, and for a time, these four taught the only courses in the law school on constitutional law, administrative law, international law, and even Roman law. See Glendon Schubert, The Future of Public Law, 34 Geo. Wash.L. Rev. 593, 594 (1965-1966).

In the first presidential address to the APSA in 1904, Goodnow discussed the subject matter of political science, including one of the “most important” subjects, public law. So how would law be studied by political scientists? How would it be different than the study of law by lawyers? According to Goodnow:

Unless we conceive of all law as a part of Political Science, it becomes necessary then to differentiate Political Science from legal science. Strictly speaking, of course all law which does not affect the relations of the State and its officers is to be assigned to legal rather than to Political Science.... But it is none the less true that a knowledge of the private law is necessary to one who would understand the methods and operations of what are known as political bodies. Furthermore, the points of contact between the private and the public law are so many and the contact is often so close that a knowledge of the private law is really necessary to one who would be a sound public lawyer.... Finally, many of the rules of private law are adopted largely because of their influence upon social and political conditions.... For these reasons the American Political Science Association has included among the subjects which are not foreign to it comparative legislation and historical and comparative jurisprudence, whether that legislation or jurisprudence is public or private. It will probably be true, however, that the distinctly private legal subjects will not receive at our hands any very exhaustive treatment.

Frank J. Goodnow, The Work of the American Political Science Association, 1 Proceedings of the Amer. Pol. Sci. Assoc. 35, 40-41 (1904).

Tomorrow’s Great Moment: Felix Frankfurter argues for more of a division between law and political science.

(Also, for an earlier discussion on the impact of Pritchett and Legal Realism on the Great Divide, see here.)

Blog Forum – Interdisciplinary Ignorance and the Great Divide

This week’s blog forum takes up the relationship between law and political science. Our guests for the discussion are Frank Cross of the University of Texas and Gerald Rosenberg of the University of Chicago. Several years ago, Cross (1997) and Rosenberg (2000) both expressed similar concerns about the gulf between legal scholars and political scientists:

“The attitudinal model ascribes judicial decisions almost entirely to politics, not precedents. To date, legal scholarship has been remarkably oblivious to this large and mounting body of political science scholarship on courts. Some political scientists have been correspondingly unconscious of the legal model.”  Frank B. Cross, Political Science and the New Legal Realism: A Case of Unfortunate Interdisciplinary Ignorance, 92 Nw. U.L. Rev. 251, 252-53 (1997);

“Today, legal academics and political scientists inhabit different worlds with little in common. If they communicate at all, they can barely hear each other; they stand on opposite sides of a great divide, and they are looking in opposite directions.” Gerald Rosenberg, Across the Great Divide (Between Law and Political Science), 3 Green Bag 2d 267, 267 (2000).

We are very pleased that Cross and Rosenberg have offered to revisit this topic on the ELS Blog. Much has changed in the past few years. Empirical and interdisciplinary scholarship would seem to be a growth industry in the law schools, which suggests the unfortunate situation identified by Cross and Rosenberg may be improving. So is interdisciplinary ignorance giving way to understanding? Is the great divide narrowing? Stay tuned.

Grand finale

I think we are drawing to a close on our blog forum, and I think Jeffrey and Bill have graciously agreed to let me have the last word.  I greatly appreciate all the points and comments, and am (naturally) particularly fond of Bill's robust defense of teaching evaluations. 

As for the question of the complexity of the relationship I am actually planning to use the underlying data for two additional projects that may reach some of Jeffrey's concerns.  The first is a large scale regression analysis where I greatly increase the number of variables to include any publicly available AALS information on the law professors studied (such as gender, years in teaching, years in practice before teaching, law school attended, primary teaching area, etc.) to see if any of those variables correlate with teaching or research.

The second is a qualitative study of the professors.  I will start with a mailing inquiring about their own views on the interaction between teaching and research, and then probably follow up with some phone interviews.  With these additional data and observations I will hopefully be able to offer more than just the raw numbers.  Thanks again for the forum, and please don't hesitate to email any further feedback or suggestions.   Ben Barton.

Teaching - Writing Correlation

There is a lot of research on student evaluations, perhaps two-thousand articles. One meta analysis indicates that student ratings are affected much more by expressiveness than by content, while student achievement is affected more by content than by expressiveness.  See 'Educational Seduction', by Abrami, Leventhal, and Perry, in Review of Educational Research. That does not mean there is no correlation between ratings and effectiveness, but it does mean we need to be cautious in making conclusions about perhaps subtle effects on effectiveness when our data are from ratings. I am not criticizing Benjamin for using ratings; indeed, I admire and applaud his effort. I would just prefer that the conclusions be stated without the logical leap. What is wrong with saying, "I found no correlation between writing output and student ratings." That would let the reader decide how much to conclude about teaching effectiveness.

As for Bill's market point, I am not arguing that these educated consumers we call students do not know what they want or that they do not know it when they see it.  But I have little faith that they want what we educators think they need, as opposed to entertainment or bar passage. They could rationally choose to give up some small bit of lawyering effectiveness in the future for some entertainment now. I have even less faith that they want to learn what society needs them to learn for the good of all. I'd bet big money that I could raise my student ratings and lower the quality of my Property instruction by replacing future interests with more on landlord-tenant law.

The more important point that I wanted to make is that Benjamin seems to be looking for a simple relationship when a complex one is more likely. To carry my optimization point further, suppose that most faculty members have experimented with more or less research time, or in some other way have found a mix of preparation and publication that maximizes the value of their own efforts, in their own eyes of course. It seems entirely possible that asking them to do more writing would result in substantially lower quality teaching, otherwise they would have made that trade themselves. On the other hand, asking them to do substantially more class preparation would result in less writing without substantially advancing their teaching effectiveness. We should not be surprised to find that each faculty member reaches his or her optimum at a different mix of teaching and writing. This sets up what might be viewed as a restriction-of-range problem. It is possible that the data would not reveal any correlation for the group even though a complex relationship would indeed exist for each faculty member.

In short, it seems likely that writing and teaching are complements at low levels of writing and substitutes at high levels of writing, and determining whether any particular faculty member has increased his or her writing to the point that further increases will reduce teaching quality is a very tricky business.

Jeffrey E. Stake

In Defense of Teaching Evaluations as a Valid Measure of Quality

Drawing upon a large representative sample from 19 ABA-approved law schools, Ben Barton’s study found no statistically relationship between various measures of scholarly output and teaching effectiveness.  If this result is correct, then much of the legal academy’s discourse on “law school quality” is fundamentally flawed.

Rather than let go of old and comfortable ideas, many law professors will look for a basis to dispute the study’s findings. As noted in Ben’s last post, the critique of his study that will probably enjoy the most currency is the claim that law students are incapable of evaluating effective law school teaching.  As a result, teaching evaluations are an invalid measure of instructor quality.

In Ben’s defense, I believe he has ceded too much ground.  There are at least three reasons why the critics of teaching evaluations are wrong. 

Teaching Evaluations

Thanks so much to Jeffrey for his extremely thoughtful response to my study.  There is a lot there, so I'll just respond in chunks, and start by ceding his conclusion: "the finding of no correlation does not eliminate the possibility of a complex causal connection."  That is likely true of any correlation study, and is certainly true here.  Any correlation study is only as good as the data, and even with perfect data a correlation study can only report a raw finding of either positive, negative, or no correlation.  As I note in the paper it is dangerous to extrapolate too much from a correlation study, and I certainly agree that I can't rule out some hidden complex causal connection.

Nevertheless, I do feel obligated to mount a defense for the much maligned student evaluation.  I have three basic lines of defense.  The first is practical.  For better or worse, teacher evaluations are the only viable way to measure teaching effectiveness for a study of this breadth.  My other choices were exceedingly unpalatable: 1) attempt to gather peer evaluation data, which is rarely if ever expressed numerically, and would also almost certainly not be provided by the host institutions; or 2) use some type of personal subjective measure of teaching effectiveness, potentially requiring me to personally visit classes and make my own determination on teaching effectiveness.  In sum, these data are not only the best data, they are the only usable data.

The second is empirical.  While there are various studies showing biases in student teaching evaluations (although I am not aware of a study that shows the biases Jeffrey mentions), there are other studies showing that student evaluations are reliable and valid, and that student evaluations correlate with other measures of teaching effectiveness, including  peer evaluations and tests of student retention.  My favorite overview is Herbert Marsh's 1987 publication "Students’ evaluations of University Teaching: Research Findings, Methodological Issues, and Directions for Further Research."   Theall and Franklin have a more recent book called "The Student Ratings Debate" that provides a good overview of the controversy, and a chapter entitled “Looking for Bias in all the Wrong Places" that updates the state of the empirical studies and provides further support for student evaluation validity.

My last line of defense is, sadly, anecdotal.  It was not that long ago that I sat in a law school classroom and evaluated my law school professors.  At that point I had experienced approximately 17-19 years of schooling, and I think I was a pretty decent judge of good teaching.  I had certainly seen my share of both over the years.  I was also poised to take on the responsibility of becoming a lawyer and representing actual clients in court.  Nevertheless, many members of the legal academy assume that I was unqualified to determine whether the teaching I received was effective, and even guess that I filled out my forms based on a popularity contest, or the ease of the course.  This strikes me as somewhat paternalistic, and an underestimation of the abilities, sophistication, and discernment of our students.  I will also note, for the record, that I am a solidly middle of the pack teacher at Tennessee based on my evaluations and I accept that as a fair evaluation.

Thanks so much

First of all, thanks so much to ELS for taking the time to have a blog forum on my empirical study of the relationship between teaching and scholarship at American law schools, and for Bill and Jeff for agreeing to comment on the study.  I can say at the outset that there were many times over the last three years that I wondered whether I would ever make it to the finish line on this project.  The biggest problem was gathering the teaching evaluation data.  The University of Tennessee's data is publicly available on line (although in a very discrete and hard to find location), so I thought I would have little difficulty gathering data from 19 other American law schools.  As it turned out I needed to contact every law school dean and associate dean, and then file 12 State FOIA requests before I reached 19 participating schools.  I had originally aimed for 20 schools, but I would have had to file a law suit to make 20, so I called it a day at 19.

The difficulties I faced in gathering the data actually helped prepare me for my counter-intuitive finding of no correlation, and also for some of the critiques I've received.  As you might guess, anyone who thinks that teaching evaluations are bunk has little use for the study.  As you might also guess, these same folks have not been shy in sharing that opinion!  I've presented the paper at 3 of the participating schools and the SEALS conference, and have gotten some great feedback.  Nevertheless, I always face at least a partially skeptical audience.  Thus, I am most interested to hear what Jeff and Bill (and any Blog readers or commenters) have to say on the subject.  I am also quite interested in how people view the ramifications of the study.  If it is true that teaching and research are not correlated, what (if anything) does that tell us about law school hiring, management, and rankings? 

Can We Measure the Relationship Between Teaching and Scholarship in Law Schools

Today, the ELS Blog is pleased to host a one-day forum on the relationship (or lack of relationship) between teaching and scholarship in law schools.  The forum is prompted by a recent empirical study by Benjamin Barton of University of Tennessee Law, which has received a lot of play in the blogosphere (see, e.g., Comglomerate, Prawfsblawg, Legal Theory Blog, Leiter, and Stuart Buck).   The buzz, of course, is focused on Ben's primary finding that there is no correlation between effective teaching and scholarship.

This topic reflects a long-running debate within the legal academy, most of which occurred without data.  In 1998, Jim Lindgren and Allison Nagelberg began to fill this void with a small scale study based of faculty from three relatively elite law schools which documented a modest positive correlation (.20) between scholarly productivity (as measured by citation counts) and teaching (as measured by student evaluations).  See Are Scholars Better Teachers? 73 Chi. Kent L. Rev. 823 (1998).   Barton's study is much broader in scope.  It includes four years of teaching evaluations for 623 faculty members at 19 ABA-approved law schools.  In addition, it includes five discrete measures of scholarly output:  academic-oriented books and articles; practice-oriented books and articles; total publications; citations per year; and total citations.

As Ben notes in his conclusion, the lack of a empirical relationship between teaching and scholarship creates quite a quandary for legal education, especially as law school competition increasingly takes the form of attracting and retaining more productive scholars.   

To probe this issue further, the ELS blog is pleased to welcome Ben Barton, who will open the forum with some additional remarks on this research.  To provide commentary on Ben's study, we also welcome Jeffrey Evans Stake (Indiana Law), who is particularly well-suited for this forum.  In addition to his innovative work on law school rankings (see, e.g.,  81 Ind. L. J. 229 (2006); 80 St. John's L. Rev. 301 (2006)) and the Rankings Game website, Jeff is an award-winning teacher who has done extensive service work to improve the teaching evaluation process. 

Realism and Institutional Incentives: A Postscript on the NLR Forum

Over the last several days, I have been thinking a lot about last week's NLR forum.   I am sympathetic to the NLR agenda of (a) pluralism in research methods, and (b) greater awareness within the broader law review literature of "law in action"--i.e., empirical insights that ensure that proposed reforms have the desired effect.  See Stewart's comment to his own post.   Further, I share Beth Mertz's goal of building interdisciplinary communities. 

That said, I don't think it is enough to conclude that these are good (or even great) ideas.  Indeed, these initiatives are remarkably similar to those advanced by proponents of sociological jurisprudence and legal realism during the late 1920s and early 1930s.  Yet, in the intervening decades, law and the social sciences developed on largely separate tracks.  The $64,000 question is "why?" 

To throw this issue into sharper relief, I want to focus on the Elena Kagan's  (Dean of Harvard Law) quote in Stewart's first post: 

The practice of law and the study of law are growing farther apart, to their mutual misfortune.  Fewer law professors understand what it means to be a practicing lawyer or what issues lawyers are confronting.  And fewer judges and lawyers look to scholars for insight or guidance.  Less and less does practice inform scholarship, nor does scholarship improve practice.

Assuming arguendo that this assertion is true, it is problematic that the "Growing Disjunction" (Harry Edward's term from his famous 1992 Mich L Rev article)  is growing at the same time that ELS and NLR are coming into vogue.   Unless these movements are fully theorized, including how they will deliver a payoff to law students, practicing lawyers, or society at large, can we really expect them to have a lasting impact (beyond padding our own c.v.'s)?

To my mind, the institutional incentives for resolving these issues are suboptimal, just like at Yale and Columbia at the beginning of the first realist wave.  See Schlegel (empirical studies lacked support because results were often at odds with realist policy agenda); Kalman (lack of funding and pay freezes at Yale, which stifled necessary appointments of social scientists); William O. Douglas, Go East, Young Man (1975) (Columbia law and social science curriculum on late 1920s blocked by law colleagues who would have to revise their teaching notes).  In an exchange in the comment to Beth's initial post, Beth queried what I meant by institutional incentives.  Basically this:  The incentives to sustain and grow NLR, as outlined by Stewart above, are at odds with individual incentives for law professors and law schools within the current law school hierarchy.

Building Interdisiplinary Communities

This is our final day of posting for this blog forum (of course, we’ll keep chatting with folks who show up over the weekend and beyond in response to things we’ve posted this week). Thanks so much to everyone in the ELS “community” for inviting us, and for this interesting discussion. I hope that we will continue to think together about many of these issues in the future.

And, speaking of communities, our topic for today is “building interdisciplinary communities.” Some of the things emphasized by NLR (combining methods, and careful translation among disciplines) go hand-in-hand with community-building across disciplines. We invite readers to chime in today with examples from their own experiences -- anecdotes about what has helped in building bridges across disciplines. (Empirical studies are of course also welcome!) Can you remember a time when you felt that you suddenly (or not so suddenly) understood what someone  from a different disciplinary background was trying to convey? Or can you remember conferences or workshops that you felt transcended the usual format of “author presents and audience critiques” and blossomed into a wider, fruitful discussion across varying viewpoints or positions?

In keeping with the NLR focus on building from the “ground level” up, I’ll take the plunge here by giving some examples from my own experience, hoping that some of you will pitch in as well. I have had the good fortune during my academic career to be a member of three truly interdisciplinary communities. As a postdoc, I spent time at the Center for Psychosocial Studies, an interdisciplinary think tank that integrated scholarship from the fields of psychology, sociology, history, anthropology, linguistics, philosophy, social theory, and law. I was then hired at the American Bar Foundation, internationally known as a leading sociolegal research institute – and a place that courageously bridges some seemingly intractable disciplinary divides. And, most recently, I was also invited to join the legendary law-and-society community at the University of Wisconsin Law School, which has steadfastly integrated social science into legal research and training all along regardless of the current fashions in the legal academy. (Okay, so I obviously love and believe in these places; sorry for the plugs – I’ll move on to more substantive comments--)

My first lesson in what it takes to be truly interdisciplinary came as a postdoc at the Center.  We were required as postdocs to attend several lengthy seminars each week. The seminars were led by advanced scholars in various disciplines; we were expected to slog through core material from different fields. Every time we encountered a new discipline, we went through several months of wondering how anyone could take this kind of research seriously. But we had a norm of maintaining inquiring rather than combative discussion, and over time we would gradually come to grasp the new field in a different way. We didn’t all immediately agree with or integrate, say, Vygotskian psychology, or Silversteinian linguistics – but we at least gained a better sense of why someone might approach the world that way. Some of us did attempt to build more synthetic approaches, both methodologically and theoretically. I think that it helped to have a sustained engagement; it helped to wrestle with core texts rather than simply presenting our own research; and it helped that we had developed respect for one another as colleagues. This collegial connection kept us working away at the translation effort at times when we were tempted to take the easier path of retreating to our own favored disciplinary stances. It can also help to have the same social issue or problem on the table, watching to see what different aspects of the problem are noticed when we employ diverse disciplinary technologies.

This idea of sharing a topic area is one of the core sources of integration, I think, at the ABF; we all do research on law, from varying vantages. There are a variety of ways in which the ABF scholars learn from each other (and I’m hoping someone else from ABF will chime in as well, since I’m sure we have different vantages on this). For me some of the most intense experiences have been the discussions we had at LSI, when I was co-editor with people from other disciplines (economics, sociology, history, psychology, law/politics). Again, a shared collegial mutual respect helped us to find ways to discuss (and even agree about) articles from our quite diverse fields. We were also driven by a desire to achieve a good interdisciplinary mix for the publication; I remember working hard to encourage submissions from good scholars from a broad variety of fields, some of which started from very different premises than my own. I have also learned a great deal at our ongoing ABF research seminars, and from occasional collaborations with colleagues from other disciplines (who are happily, right down the hall).  The ABF philosophy has been that interdisciplinary work in law will flourish best if it is grounded in cutting-edge social science research which can meet the standards set by social science fields themselves.

I’m still getting to know the community at Wisconsin, which is very rich in its commitment to studying law “on the ground.” One thing I’m learning is that it takes effort to keep the discussion going. My colleagues work hard to keep finding ways to communicate about everything from teaching issues to research and policy questions. Junior faculty members who do not have social science training have gone so far as to take multiple classes in social science methodology on campus (taught by their dual-degree colleagues!). The Institute for Legal Studies (under the directorship of Howie Erlanger) stands willing to organize the equivalent of a doctoral committee from across various departments and the law school, in order to review project proposals and to give feedback on ongoing law faculty research. So, one NLR idea: find a way to create sustained, ongoing discussion, centered on a specific research problem, and bringing together faculty from different disciplines. This can be done at many universities without even leaving campus. Wisconsin's senior faculty also invests in sociolegal research by supporting junior scholars -- giving them research leaves, reading their work, etc. (and you can pretty much count on being asked about how this plays out in practice, whether you are junior or senior!)

Of course, as this blog demonstrates, we also now have all the tools given by the internet and new forms of communication for building interdisciplinary communities. It will be interesting to watch how these tools can best be put to work to help us achieve more integrative empirically-grounded approaches to studying law.

 

Legal Realism, Herman Pritchett, and the Great Divide

Barry Friedman’s “Taking Law Seriously,” 4 Perspectives on Politics 261 (2006), calls not only for taking law seriously but also for improved collaboration between political scientists and legal scholars, which is likely critical for better incorporating law into the empirical work on the courts. (See Michael’s earlier post here.) Some of Friedman’s concerns have been with the field for a long time, but even these bear repeating. See, e.g., Robert G. Dixon, Jr., Who Is Listening? Political Science Research in Public Law, 4 PS 19 (1971). Rather than focus on this larger theme, I want to offer a few thoughts on his historical comment about how we got to where we are today, a topic that ties in both with “New Legal Realism Week” and with “Interdisciplinary Ignorance and Division Week,” which is scheduled for October.

Friedman summarizes the origin and development of the divide between political scientists and legal scholars as follows:

Familiar history recounts how, beginning with Herman Pritchett in the 1940s, political scientists interested in the courts went their separate way from scholars in the legal academy. Legal realism occasioned the split. Legal realists recognized that judges, being human, were likely to see legal issues through the lens of pre-existing social and political commitments. Legal scholars, troubled by what Realism spelled for law’s legitimacy, took a normative turn, devoting their efforts almost exclusively to telling judges how they should decide cases. Political scientists, on the other hand, fascinated by Realist claims, chose the positive path, seeking a better understanding of how judges actually decide cases, and why.

Friedman at 263. Friedman does not develop this history in any more detail. In fairness to him, his essay is not meant to be a history lesson. But this short summary strikes me as problematic.

Taking Social Science into the Law School Classroom

Greetings, folks. This is turning out to be great fun for us; we are having quite a few discussions off-line with colleagues as well as on the blog. Thanks to everyone who is joining us one way or the other!

Today we are hoping to generate some further discussion on the issue of integrating social science into legal pedagogy. A number of NLR scholars have echoed Bill Henderson’s earlier sentiment that it’s very important to pay attention to “how realism affects the education and training of law students.” Bill’s previous set of questions on this point challenged us to consider how to integrate our roles as scholars and law teachers.

In reply, I drew on my forthcoming book, The Language of Law School: Learning to “Think Like a Lawyer”. My study was based on transcripts of actual law school teaching. We taped all of the classes from the entire first semester of Contracts classes in eight very different law schools, and then analyzed them qualitatively and quantitatively, turn by turn. I found that professors whose research is frankly critical of doctrinal approaches send linguistic messages in first-year classes that are in some respects quite similar to those sent by professors who are doctrinal scholars. This can be traced through subtle aspects of the language cues found in classes which differ in other ways (predominantly lecture vs. more conversational vs. “modified Socratic” styles). These language cues tell students to reorient their attention from social context to the layers of legal authority associated with doctrinal readings.

On the one hand, there’s a real dilemma here. Law professors cannot in good conscience send their students out into practice without the ability to parse doctrine. This is a new skill when students enter law school, and it is a major focus of first-year teaching. Students need to take stories of social conflict and read them in a very different way, disregarding some elements that would have seemed crucial before. Once they have learned this kind of reading, they have incorporated a new sense of what’s important, where authority rests, where their attention belongs. This reorientation is not so easy to simply discard or put aside. If we as scholars are having trouble integrating the epistemologies and norms of law and other disciplines or perspectives, what can we expect of our students?

On the other hand, people can be bilingual. Certainly anyone who becomes a trial lawyer will have to relearn a more common kind of narrative convention. Bob Burns, author of A Theory of the Trial, recently told me that as a clinical law professor he often feels that he has to retrain law students to understand and tell “normal” stories of social conflict – to “undoctrinate” them to a certain degree. So, can we also train them to read and understand social science of various kinds, and to integrate that with their roles as lawyers? (One NLR response would be that maybe we have to think through that issue of integration ourselves if we are to teach it to our students.)

My own personal experience in law teaching is that empirical research can serve as a wonderful bridge between “practical” and formal legal training. Research on law “on the ground” can be immediately appealing to students who long for some sense of what their lives will actually look like once they leave law school. This is particularly true of research on the legal profession itself. John Conley, a fellow anthropologist who teaches at UNC, has carried on a very interesting class at the law school there for a number of years: students interview practitioners as part of their class on the legal profession, in the process both creating an ongoing data base while learning about the practice of law.

I recently attended a very interesting conference at Vanderbilt on legal education (at which there was quite a bit of discussion about integrating empirical research and legal education). I was particularly struck by a distinguished practicing attorney who spoke to us about his wish that law professors would stop conveying a sense of disrespect and disdain for the practice of law to their students. He didn’t want us to stop educating them about all kinds of interesting and different perspectives on law; when else will they have that important opportunity? But he was asking why there has to be such antagonism across parts of the profession. Arguably,  empirical research that takes law on the ground seriously could help law professors walk in the shoes of those who are struggling with the realities of law practice. (Of course, it could also help lawyers walk in the shoes of their clients as well – here ethnographic and interview data might be quite useful.) Louise Trubek has suggested some intriguing ways to erode the boundaries that separate clinical education, interdisciplinary training, and classic law school pedagogy. (“Crossing Boundaries: Legal Education and the Challenge of the ‘New Public Interest Law,’” 2005 Wisc. L. Rev. 455.)

I do work with my students on how to judge relevant social science information when I'm teaching  family law, asking them to assess critically studies purporting to show the effects of divorce on children, the effects of mediation on outcomes in divorce, etc., etc. They will be receiving this information one way or the other, through the popular media or in CLE training. I think it is quite possible to integrate some training in social science method into the core curriculum. But we still face the question of how to help our students navigate the difficult cognitive and linguistic clashes that arise when they grasp the real discontinuities between legal and social science thought. We can’t ignore the fact that we aren’t training would-be social scientists.  There is likely a limit in the amount of energy someone who plans to practice law is willing to invest in learning the details of social science methodologies.  And so we have a responsibility to think through how best to take account of law students' quite different professional trajectories when working to integrate our kind of research into law training.

We can also use the exiting empirical research on education to improve how we teach. As Peggy Davis and others have noted, this is a very interesting aspect of the relationship between social science and law teaching -- one about which we don’t often speak.

 

 

Leiter on New Legal Realism

Brian Leiter, who is an expert on the original legal realism (see, for example, this article), has been reading our NLR forum.  He has some harsh things to say ... and they are worth reading. 

I am not an expert on legal realism, nor do I intend to become one.  But I have been reading quite a bit on the topic because I think it is highly relevant to understanding modern legal scholarship and the "industry" of legal education--i.e., it's crucial background.

Brian offers the most succinct rationale I have read for the pervasive influence of legal realism and why we shouldn't lament the failure of the original realism to develop an empirical component in the tradition of the social sciences:

The actual Legal Realists ... paid homage to the social sciences, even adopting the rhetoric of the then-dominant behaviorism (e.g., talk about the "stimulus" of the facts of the case), but their actual scholarly practice was almost entirely insulated from the social science of the day ... .

This isn't to say that the Realists weren't interested in what the courts do in fact, it's just that their approach to the facts about what courts do almost entirely eschewed social scientific inquiry, and for good reasons I think.  The paradigmatic Realist inquiries of the 1920s and 1930s [citing lots of examples] ... consisted in careful scrutiny of the underlying facts of lines of cases, bringing out the gap between the official "doctrinal" explanation for the decision and the actual sotto voce norms that seemed to be at work in the judge's thinking.  The goal was to discover the non-legal norms that made best sense of the courts' response to recurring "situation-types," i.e., patterns of fact that seemed to elicit the same kind of results.  ...

That the preceding was the heart of the Realists' "empirical" method explains, of course, why the Realists were so influential in American law:  you didn't need social science training to do this kind of analysis, you just needed to be a sensitive and skeptical reader of court opinions, something good lawyers are, well, good at.  ...

A "new legal realism" would continue the paradigm of scholarship established by the old legal realists, namely, contrasting what courts say they’re doing with what they actually do.  "We are all realists now" because this is what so many legal scholars do, including those who know nothing of social science and don't even self-identify as realists.

These are interesting and important points.  From my own perspective, NLR is really about developing a theoretical framework (or a workable consensus) on how and when to utilize relatively new research tools and perspectives; and this requires a renewed engagement with the social sciences, which have changed a lot in theory and method since the 1930s.   As Stewart noted in a comment, we should not overinterpret the name "new legal realism".   Picking the name merely started a worthwhile conversation.  If the conversation continues over a period of years, the NLR label will grow legs.

Combining Quantitative and Qualitative Methods in Socio-legal Research

In their opening comments and responses to other posts, Beth Mertz and Stewart Macaulay ably made the connection between the New Legal Realism (NLR) and established traditions of empirical inquiry among law and society scholars.  Stewart reasserted a theme of his own classic work, that legal scholars should seek "a more accurate picture of law in action" before they offer proposals for changing law.  Beth called attention to the need for socio-legal scholars to more self-consciously translate their research findings to have a broader effect within the legal academy, among policy audiences, and among the public.   Both  advocated  a multiplicity of  empirical methods,  a self-critical humility for the nature of the claims that can be supported by different modes of empirical research, and at least an implicit recognition of the need to connect empirical inquiry to broader theoretical propositions about law and society.  As empirical legal studies expands and matures in the legal academy, these principles are important for the intellectual significance of the enterprise. 

In this essay I want to briefly argue that theoretically driven research that uses multiple methods can produce stronger validity claims, can better illuminate the social mechanisms through which law operates, and may lead to research findings that more readily translate to broader publics.  In other words, research that advances the goals of NLR.  I will try to carry the argument with the examples of three research projects, one by a colleague and two in which I have been personally involved.

Laura Beth Nielsen's License to Harass (Princeton University Press 2004) studied offensive public speech through observation and in-depth interviews with some 100 respondents, including white men, white women, and men and women of color.  She found that the experience of these groups with sexist and racist speech on the street varied dramatically by social group, with women and people of color encountering harassment at shocking levels.   While all social groups  saw  offensive speech as a serious social problem,  a large majority of respondents in all social groups opposed legal intervention to deal with it.  But the groups based their positions on different rationales.  White men articulated a classic first amendment view, women invoked a discourse of autonomy (they would take care of it themselves), and people of color (especially men)  spoke of a distrust of authority (they thought the law would be used against them).   Without an in-depth investigation of the discourses employed by these different groups, we would know much less about their legal consciousness concerning offensive public speech.  Nielsen is thus able to offer some profound insights into the experiences of different social groups on the street, their attitudes about the law, and about the role of law in shaping those attitudes and providing a license for offensive speech to continue to operate.

In Legalizing Gender Inequality  (Cambridge University Press 1999), William Bridges and I examined four case studies of litigation concerning gender-based pay discrimination.  Because the cases involved comparisons of wages across different predominantly male and predominantly female jobs, the cases were characterized as comparable worth cases.  The defendant employers in these cases invoked a market defense, arguing that the wage differentials reflected market processes rather than gender discrimination.  The courts in 3 of the 4 cases accepted that defense.  Our book subjects those legal holdings to empirical scrutiny through a variety of labor market, employment payroll data, organization process data, and in-depth interviews.   The different forms of data converge to reject the market defense.  For example, in AFSCME v. State of Washington,   we found that a state pay survey that was treated as a neutral scientific instrument for determining job-level pay was in fact riddled with arbitrariness and bureaucratic politics.  As a state labor union representative commented, they were not "going to rob Peter to pay Pauline."   The book concludes that the courts in these cases  interpreted antidiscrimination law as  potentially providing a remedy for between-job pay discrimination,  while at the same time refusing to critically analyze actual wage-setting practices in these organizations.  The result was  that the courts legitimated between-job gender inequality in pay.

The final example concerns a project currently underway by Laura Beth Nielsen, John Donohue, Peter Siegelman, and myself.  The project examines the changing dynamics of employment discrimination litigation from 1987 to 2002.  We possess a full set of EEOC complaints from that period, in addition to a large random sample of federal court filings from 7 jurisdictions, which we have coded and begun to analyze quantitatively.  The statistical data will allow us to comprehensively  examine  the system  of discrimination disputing and to test whether it is operating differently than  during the 1972-1987  period analyzed by Donohue and Siegelman in their earlier work.    Employment discrimination litigation has become an intensely contested arena.  Many commentators assert that the dramatic rise in discrimination lawsuits, which roughly tripled between 1991 and 1998,  is a result of growing numbers of frivolous lawsuits.   

We are  also interviewing in depth parties and their lawyers  in a selected subset of  cases.   After conducting some 50 in-depth interviews thus far, we already have gained important insights into how the system operates.  The interviews have literally brought these cases to life.   One African-American plaintiff described how he had been asked by his immediate supervisor whether he had ever had sex with his daughters, as he had heard that slaves used to do.  When internal complaints did not remedy his work environment, he filed suit.  In our dataset he is coded as an early settlement on the outcome variable.  But what the coded file does not indicate is the gravity of the insult he received, the fact that as a result of the lawsuit he became divorced, went bankrupt, felt enormous physical stress.  Although he won his job back as part of the settlement, he was laid off in a reduction in force a year later.  When asked whether he would file the lawsuit again, he said no, that he "would have took it."   In this case, as with almost all the plaintiffs we have interviewed, we find serious injuries, even though the proof of illegal discrimination sometimes is shaky.  Indeed, the defense attorneys largely concede that most of the cases they encounter involve individuals who have been "wounded" at work. 

While we have more data to collect and analyze, the combination of systematic statistics and in-depth interviews is likely to produce a better understanding of antidiscrimination law in action than presently exists.   This combination should allow us to disseminate our results more effectively to legal, academic, policy, and public audiences than might be the case with only qualitative or quantitative data.   

My take on new legal realism

    One obvious question about New Legal Realism is how it might differ from the old.  Can we claim anything more than we have a website, while Karl Llewellyn didn't?  (See http://www.newlegalrealism.org).  As Bill noted in his Introduction to this blog forum, much of the old legal realism was aimed at freeing appellate judges from the confines of formal approaches so that they could carry out good policies or their "situation sense" as they decided cases.  A new legal realism is not as focused on appellate judging as the old.   Instead, new legal realism takes full advantage of about forty years of law and society research that suggests what the law in action looks like.

    I've commented elsewhere that we would be making progress if mainstream legal scholarship would simply take account of this research.   Although participants in this blog are familiar with such points as law is not free and law is delivered by actors with limited resources and interests of their own, too much writing in law reviews still proceeds with little concern with these well-documented realities.   In a talk I gave at the contracts panel at the last AALS meeting, I played with an analogy to the fate of the nuclear submarine USS San Francisco.  In January of 2005, it was cruising at top speed at a depth of more than 500 feet.  It crashed into an undersea mountain, killing one sailor and injuring 97 others.  The crash caused more than $88 million damage to the boat.  The mountain was not on the boat's navigational charts, although potential hazards in the area were noted on other charts that the San Francisco's officers had never seen.  Legal scholarship that proceeds without awareness of the existing empirical charts of the law in action risks a  similar fate.  I did conceded that contracts scholarship is unlikely to kill or injure anyone.   Nonethelesss, it could be rendered ineffective, or even have unintended consequences,  if it is not based on an accurate picture of the law in action.   And a major part of that picture involves negotiation and discretion rather than a neat rule-bound system.

    How do we get a more accurate picture of the law in action?  We must turn to the methods of all the social sciences.  Law is too complex to grasp fully if we favor one approach over all the others.  My late wife Jackie Macaulay had a Ph.D. in social psychology from a department that prided itself as the "Dust Bowl of Empiricism."  (She later became a lawyer).  She suggested that the best working rule was to presume that all social science approaches are flawed, but the game is to understand the limits of each and use the best approach available for what you want to study.  Sometimes we can get neat tables of numbers and use state of the art statistics.  She did some of this kind of work herself, but she always suggested that I ask where the numbers in the tables came from.  Human behavior doesn't turn itself into numbers by some magical or natural process.  Someone has to code it, and the quality of the coding varies.  Many small, unfounded assumptions can be hidden within apparently neutral quantitative results.

    Our Wisconsin colleague Bert Kritzer compared what he learned from asking questions with what he learned from watching lawyers in contingent fee practice.  He says,  "the observational studies present more nuanced images, which tend to contradict the more straightforward results of the interview-based analyses."  He quotes Bob Dingwall:  "interviews construct data, observers find it;  in an interview study, we can pick and choose the messages we hear and that we elicit, [while] in observation we have no choice but to listen to what the world is telling us."  Of course, often it isn't easy to get permission to watch, and even when it can be done, it is a much more time-consuming approach.  And, as with all other methods,  there is a discipline to observation which, if ignored, can lead to unreliable results.

    Sometimes we are limited by the available sources of information.  We may not be able to interview a randomly selected sample.  There may be a limited number of people who know what is going on and who are willing to talk to a researcher.  Moreover, often such people are unwilling to talk if you do not promise confidentiality.  There are, of course, a number of problems when we go down this road.  For example, our informant may tell lies or spin his answers to make himself look good.  She may seek to entertain us by telling us about the atypical or interesting events rather than the dull everyday practice.  It is hard to know how representative our informant's knowledge is.  If we were able to talk to more people with different experiences, would the story differ significantly?  And other scholars cannot replicate our work because we cannot tell them who we talked to.   I've seen some suggest that we should not interview business people or lawyers about contracts practices because we cannot get a good sample and our work cannot be replicated.  I'm sure that my reaction is influenced by the alternative -- often if we don't rely on those informants we can get, we can do nothing.  I'd prefer to recognize the limitations of such approaches and flag them in my articles.  For example, in my article on Lawyers and Consumer Laws, I described the research that involved interviewing about 100 lawyers in five Wisconsin counties as well as its limitations.  I concluded:  "It [the article] should be read as a report from a preliminary study, offering suggestions that the author thinks are true enough to warrant reliance until someone is willing to invest enough to produce better data and lucky enough to find a way to get them."   You can take some steps to minimize the risks, and we should where we can.  At the minimum, we can challenge the story that we get and ask whether it is plausible.   Then we can wait for other studies and see if they find much the same thing.  One lesson of new legal realism, gleaned from good social science practice, is that we need a healthy dose of humility in drawing conclusions from our research.  And this is true whatever flavor of social science we prefer.

    Dean Elena Kagan represented the new Fellows of Class III, Section 4 of the American Academy of Arts & Sciences.  She said:  "The practice of law and the study of law are growing farther apart, to their mutual misfortune.  Fewer law professors understand what it means to be a practicing lawyer or what issues lawyers are confronting.  And fewer judges and lawyers look to scholars for insight or guidance.  Less and less does practice inform scholarship, nor does scholarship improve practice."  A new legal realism would encourage academics to study what lawyers do, and with what consequences to whom.  This should be worth our attention.  My guess is that what lawyers do, and don't do, matters.  One way that we can translate more effectively between empirical research and legal practice is to gain a better understanding of the realities of law on the ground, and quite often (althought not always) it is lawyers who put the law into action.   Of course, a thorough study of law on the ground means that we not only include law on the books and law in the attorney's office, but we need to add law in everyday life.  This research into the living law has been ongoing for many years, and  law professors and students should be far more aware of it.

    What might we gain by the new legal realism project?  Law professors advocate legal rules that they hope will gain some measure of, in Llewellyn's phrase, the good, the true and the beautiful.  We write amicus briefs and law review articles and we draft statutes and regulations seeking reform.  Recognizing all of the problems with offsetting power, we should do better if our proposals reflect the way  law affects people from the ground up.   If we care about justice, we need to understand much more than the text of legal rules.  Here is where the new legal realism continues the spirit of the old.

Empirical Research and Interdisciplinary Translation

What a great introduction, Bill -- thanks!  We appreciate being invited to join this interesting conversation on the ELS Blog.  Stewart Macaulay and Bob Nelson will be my co-hosts, and together we'll try to do justice to describing the ongoing efforts of the scholars who've been involved in the New Legal Realism Project (NLR).

NLR began with a group of social scientists and law professors who were concerned about the difficulties involved in translating between law and social science.  Initial discussions were held at the American Bar Foundation and the University of Wisconsin's Institute for Legal Studies (ILS), both of which have had decades of experience in fostering the social science study of law.  The First New Legal Realism Conference, funded and organized by ABF and ILS, was held in Madison in June 2004.  Many of the papers from that conference are appearing in a unique publication collaboration between the Wisconsin Law Review, a student-edited law review, and Law and Social Inquiry, one of the leading peer-reviewed journals in the area of empirical research on law.  (Working out that collaboration turned out to be a very practical exercise in interdisciplinary translation!)  We've since followed up with a series of conferences and events, with more to come.

We started out with the observation that achieving a high standard in interdisciplinary translation requires some attention to the translation process itself.  On the one hand, legal professionals may try to pick up on social science findings without understanding the frameworks from which those findings emerge.  This can lead to misinterpretation of research results, or to reliance on faulty social science.  On the other hand, social scientists may also assume that the translation process is transparent, failing to grasp the very different epistemology and goals of legal language and processes.  I often quote Epstein and King's pithy summary:  "An attorney who treats a client like a hypothesis would be disbarred; a Ph.D. who advocates a hypothesis like a client would be ignored."  While law professors don't have to treat their favorite theories like clients, their training and professional culture can arguably predispose them to do so.  And, to the extent that they are debating normative and interpretive principles rather than empirical evidence, law professors have defensible reasons for proceeding differently than social scientists do.

Despite the many divergences among the social sciences, I do believe that social scientists have a distinctive shared commitment:  if our theories conflict with what we discover through well-established research, it is the theories and not the findings that need to be discarded or changed.  This operates as a check on researcher hubris -- on our very human tendency to see what we want to see in the data we collect.  (Dan Kahan provided a great example of this in his guest blog last week.)  It is also a check on power, because at its best, empirical research can permit our research subjects to talk back to us -- to shake and jar the assumptions and pre-given frames with which we approach them.  (And, because I was trained as an anthropologist, I have to say I think that well-done, rigorous participant observation is one of the best empirical methods for achieving this kind of check on theoretical assumptions.  But I know that other methods have their advantages as well!)

On the other hand, social science disciplines themselves differ in terms of methods, goals, and epistemological assumptions.  As Monahan and Walker explain in their classic text on social science in law, social scientists must often sacrifice precision in one part of the picture in order to obtain a fuller understanding of another (e.g., internal versus external validity).  Obviously, the more that we can combine different kinds of methods, the less we have to sacrifice.  Bill Henderson and Howard Gillman made a similar point in earlier posts on this blog.  I liked Howard's comment that "empiricism takes a variety of different forms."  As he correctly noted, excluding either qualitative or quantitative research from our definition of "empirical" is not only technically erroneous -- more importantly, it would cut off important sources of information.

To summarize, the NLR network of scholars is working to develop more informed and precise methods of translating between social science and law.  This requires that we begin by examining the issue of translation itself, as opposed to assuming linguistic transparency among disciplines.  If legal professionals are to obtain the fullest possible understanding of law in action, they will need to draw on the broad range of social science disciplines and methods currently available.  To achieve this, we will need a theory and practice of translation that is itself empirically grounded.  We will also need to synthesize research conducted using different methods  (something Bob will be talking about on Wednesday).
------------------------------
A personal PS:   Here I want to add a comment that represents just my personal take on this issue of interdisciplinary translation.  (Like ELS researchers, NLR scholars come to the table with a variety of disciplinary and other positions -- this is mine.)  As a linguistic anthropologist, let me put in a quick plug for my field.  The past two decades of empirical research on language and communication in anthropology and sociolinguistics has shed some valuable light on processes of "translation" between different institutionally-based discourses (or, in the terminology of Michael Silverstein at the University of Chicago, "transduction").  One key finding is that our ongoing ability to communicate depends in structured ways on metalinguistic cues (and, specifically, on "metapragmatic" cues that signal us about how our speech connects with context).  Thus we are likely to miscommunicate when we translate across disciplines if we do not first examine divergences in metalinguistic assumptions and structures.  (This is not unlike Kahan's warning that we need to unpack underlying cultural worldviews in order to bridge communicative divides -- and indeed, language research has connected cultural worldviews with metalinguistic structure.  In my own research, I've used transcripts of first-year law teaching to examine some aspects of legal metapragmatic structuring.)
------------------------------
There's lots more to say, and I'm looking forward to the discussion!  ABF Director Bob Nelson, whose well-respected sociological research on law has combined multiple methods, will lead off on Wednesday.  Tomorrow we'll get to hear from Stewart Macaulay, renowned for his research on relational contract and also co-editor of a classic text on social science of law.

Blog Forum - The New Legal Realism: It’s Not About Breakfast

A while back, Legal Realism became associated with an eccentric view of judicial behavior, i.e., that it is better explained by judges’ breakfast selections than anything remotely identifiable as law. As Judge Kozinski described it, “If the judge has a good breakfast and a good night’s sleep, he might feel lenient and jolly, and sympathize with the downtrodden. If he had indigestion or a bad night’s sleep, he might be a grouch and take it out on the litigants.” Alex Kozinski, What I Ate for Breakfast and Other Mysteries of Judicial Decision Making, 26 Loy. L. A. L. Rev. 993, 993 (1992-1993); see also Ronald Dworkin, Unenumerated Rights: Whether and How Roe Should Be Overruled, 59 U. Chi. L. Rev. 381, 391 n.16 (1992) (referring to “the old realist thesis that the law is only what the judge had for breakfast”); Ronald Dworkin, Law’s Empire 36 (Harvard 1986) (“Some realists ... said there is no such thing as law, or that law is only a matter of what the judge had for breakfast.”). As many have pointed out before, this is not an accurate characterization of Legal Realism. See, e.g., Frederick Schauer, The Limited Domain of the Law, 90 Va. L. Rev. 1909, 1923 (2004).

But now there’s a New Legal Realism, and we want to help start things off right, lest it too become wrongly associated with one of the big three meals of the day. According to the New Legal Realism website, the basic idea “is to develop rigorous, genuinely interdisciplinary approaches to the empirical study of law.” This is right up our alley here at the ELS Blog. We are therefore pleased to have with us this week Professors Elizabeth Mertz (American Bar Foundation & Wisconsin), Stewart Macaulay (Wisconsin), and Robert Nelson (American Bar Foundation & Northwestern). They will explain New Legal Realism, what it is, what makes it new, and why it’s not about pancakes verses waffles.

Coping with the “Petty Tyranny” of IRBs

Edward Abbey quipped that “No tyranny is so irksome as petty tyranny.” But, rather than fighting or railing against the system, my final post this week invites discussion on coping with IRBs. My prior post, and Jack Katz’s, talked about claiming exemption and when it’s safe to not even file. This post focuses on the “minimal risk” threshold. Below it, protocols may qualify for expedited review and waiver of formal consent, but beware ye who pass above it.

Richard Campbell, member of a now-defunct advisory committee, offers this excellent analysis: the Common Rule defines minimal risk as “not greater than those ordinarily encountered in daily life.” This should mean the sort and magnitude one encounters working, shopping, driving, or having a doctor’s check-up, which includes temporary or low-grade harms such as stress, anger, embarrassment, stubbed toes, paper cuts, etc. Campbell notes that risk of even disastrous harm meets the “everyday” standard if it’s fairly rare, such as an auto accident.

Building on the transportation example, Bill Ford articulated the absurdities of regulating car driving risks as we do research risks. It’s easy enough to point out differences between the two that justify regulating one but not the other. But the key point is to match the type of regulation with the degree of risk. Ford’s absurd example of filing route plans before driving is in fact how general aviation is regulated by the FAA (more or less) - which makes sense for its degree of risk. Similarly, the Common Rule, as written, could limit and modulate IRB oversight appropriately - if only IRB administrators could be convinced. In response to Ron Wright, perhaps that’s what law professors should be trying to do. 

Putting Lawyers into the IRB Mix

Most legal academics at most law schools have never submitted research proposals to an IRB. Most, I would guess, have no idea that their research probably falls within the asserted jurisdiction of the IRB on their campus.

If this benign (?) neglect disappears over the next few years -- as I think it will at many universities -- what will be the effects on practices in law schools, and what will be the effects on IRB practices? I have a prediction on the second question.

Based on other interactions between the university administration and the law faculty, I'm betting on some healthy turmoil. When the pension fund administrators at the university make claims and suggestions about personal and university investments, the law faculty start talking about securities fraud. When the university administration wants to "manage" the message coming from the college radio station, the First Amendment mavens at the law school get lots of air time.

The point is that law faculties, as a whole, have a reputation on many campuses for pushing back against assertions of administrative authority. They are feisty, even contentious. Other groups of faculty might prove just as feisty, but the law faculty neutralize one favored administrative technique -- claims about the law -- that work as debate-stoppers when it comes to other faculty groups.

I think this dynamic will hold true to form when law faculty start interacting with IRBs in larger numbers. When an IRB makes an overly broad assertion of power, based on its reading of the regulations, a lawyer is more likely than a biologist or a sociologist to reject the current way of doing business and propose something different. If, as Jack Katz mentioned in his recent post, an IRB sees its mission as preventing research that could provoke some legal claim against the university, a legal researcher will be pretty vocal in pointing out the weaknesses of such a civil claim. And what does prevention of long-shot civil claims have to do with the Belmont principles?

If you chaired an IRB, wouldn't you quietly hope that legal academics remain in a gray area of possible non-compliance, just for the peace and quiet? 

Outside of Biomed Research, IRBs are Essentially Censorship Agencies

Like many ethnographers, I often do my research by putting myself in places and relating to people without a very clear idea as to what I’m looking for.  Even when I have a clear idea, like when I’m going to interview someone about their life in a neighborhood, I’ll see things on the way there and back that may be more valuable than what I find out when my questions get answered.  Much of my fieldwork becomes “data” in retrospect.  This makes it impossible to clear much of my research beforehand with an IRB.

To different degrees, many social researchers, not just anthropologists but political scientists and many others who would create “generalizable knowledge” and thus fall within IRB jurisdiction, work this way.  So do law professors, as entries on this blog have already acknowledged.

This means we are all vulnerable to attack for violating the rules in our universities or colleges. There are ways of limiting our vulnerability; I’ve detailed some in a paper that’s part of a collection currently being considered by an anthro journal.

IRBs and Regulatory Theory

When I first saw an IRB at work, it reminded me immediately of a creature that is familiar to Administrative Law teachers: the agency that has incentives to ignore Type 2 errors in favor of Type 1 errors. The classic example is the Food and Drug Administration. The FDA knows that if it wrongly approves a dangerous drug for sale (Type 1 error), then when somebody dies after using the drug (say, using thalidomide as a sleep remedy for pregnant women), there is hell to pay. Public outrage, congressional hearings, you know the drill. On the other hand, if the FDA wrongly fails to approve -- or wrongly delays approval -- of a safe and useful drug (Type 2 error), the costs are more diffuse and less visible. There are exceptions (think of recent pressure on the FDA over AIDS treatment drugs), but the culture of the FDA pushes everyone in the agency to be most alert for Type 1 errors, even though both types of errors are costly to health.

The same incentives operate within a university IRB. The consequences of Type 1 errors -- such as wrongly approving harmful or non-complying research protocols -- are disastrous, including suspension or loss of funding for the entire institution. The IRB, on the other hand, tends to ignore the consequences of Type 2 errors, such as the delays in research, the research projects that never occur, the research that is needlessly scaled back in ambition.

This old chestnut from regulatory theory leaves a sobering message about IRB reform. It is not enough simply to point out the possible Type 2 errors and expect the IRB to account for them from now on. There are powerful reasons why Type 1 errors get the most attention. The strategies have to focus on ways to make Type 2 errors more visible and salient across many different bureaucratic cultures and across research proposals from many different parts of the academic world.

What we need are ideas that are universal amplifiers for Type 2 errors, loud enough to make stifling of research a routine worry for the IRB, on an equal footing with the Type 1 errors. Got any? 

Confessions of an Undocumented Human Subjects Researcher

I confess.  Like Jimmy Carter, I have lusted in my heart, and like most law professors, I have engaged in unprotected human subjects research (HSR).  Just last week, for instance, I called a lawyer and an insurance executive to get their views on a legal issue I was researching, and I might cite them in the article I’m writing.  Later, I wondered whether I should have had IRB permission first to do that. 

I expect that a good portion of this forum on IRBs will address overregulation, but I doubt we’ll be able to do much about that in the short term, so I would like to kick off a second line of discussion this week: practical tips for coping with IRB oversight.  To focus discussion, note that there are two regulatory thresholds: 1) is an activity non-exempt human subjects research (HSR) to begin with; and 2) if so, does it pose more than minimal risk?  This post focuses on the first of these two thresholds. 

As summarized in the IRB FAQs that Ron Wright and I posted several weeks ago, there are several bases to claim that interaction with, or data about, living humans in the course of legal scholarship is not a covered activity (not identifiable, not systematic, not private, etc.).  Naturally, those concepts have room for interpretation.  The hitch is that institutions expect researchers to notify the IRB so it can have the final say on whether the exclusionary concepts apply. But, that quickly leads to innumerable absurdities.  Every hour of every working day, I do things that are not HSR.  Which of those things do I need to tell the IRB about, simply to confirm they aren’t non-exempt HSR?  Do we also need to define a grey zone threshold for when an activity MIGHT be interpreted as covered HSR by an IRB?

Not necessarily.  I believe this threshold decision is one that researchers should be able to make on their own in good faith, albeit at their own risk of being wrong.  If I’m not seeking external funding, and I’m not publishing the research in journals that routinely deal with HSR, then it’s not likely that anyone will require me to verify the absence of HSR.  And, if in fact I haven’t done any HSR, then have I done anything wrong by not clearing the activity first with the IRB?   If not, then maybe I only need to seek IRB approval if I’m not sure about the activity being HSR, OR if I anticipate having to verify to an outsider (journal or funding agency) that it isn’t covered HSR.  Am I wrong about this?  How much trouble could I or the institution get into for doing things that aren’t covered HSR without seeking IRB approval first?

Addendum:  Jack Katz’s related post refers to an excellent paper of his (available soon here, and in press at the American Ethnologist), that develops two key points in detail:  1) IRBs are not required to review research that is not federally funded.  Although most Universities require this, some prominent ones are starting to step back from that.  2) Some types of research involving humans are not sufficiently “systematic” to meet the Common Rule’s threshold definition.  It’s a really good paper; you should read it. 

Please Have Your Consent Forms Ready

Is change afoot in the regulation of social science research by IRBs, especially low risk research involving surveys, interviews, and the use of already existing data?  One source of change could be the intensifying scrutiny of IRBs. In April 2003, the Center for Advanced Study held a conference on IRBs and published a white paper on the topic, Improving the System for Protecting Human Subjects: Counteracting IRB “Mission Creep.”  Next month at Northwestern University, Philip Hamburger (Columbia) and James Lindgren (Northwestern) are holding a conference on the First Amendment implications of IRB regulations.  If academic conferences can provoke change anywhere, maybe it’s in academia.  Another possible source of change is the increasing interest of legal scholars in empirical research, which means more IRB oversight of their work.  Presumably, regulating legal scholars increases the likelihood of an unhappy researcher filing a lawsuit against an IRB -- and perhaps making the arguments generated at these conferences to a court.  Whether changes are likely and what changes are desirable are two of the topics up for grabs this week.

During this second Blog Forum, four guest bloggers will offer their perspectives on IRBs.  Kristina Gunsalus of the University of Illinois’ Office of University Counsel and the College of Law (Urbana-Champaign) has written widely on the topic of IRBs.  She is the primary author of the Center for Advanced Study’s white paper.  Mark Hall and Ron Wright of Wake Forest University’s School of Law have experience from the law school side of dealing with IRBs.  Professor Hall specializes in health care law and policy.  Professor Wright specializes in criminal justice and administrative law.  Jack Katz of U.C.L.A.’s Sociology Department is a participant in the upcoming conference at Northwestern University.  He specializes in social psychology; ethnographic methods; urban communities; and crime, law and deviance.

Next Week: A Blog Forum on Institutional Review Boards (IRBs)

Suppose long ago the local Departments of Motor Vehicles, perhaps under the auspices of the Federal Motor Carrier Safety Administration, started regulating the transportation of hazardous chemicals on public roads. They would inquire about destinations, routes, and safety protocols. Since the risks from these materials may be significant, this oversight might be a good idea. But at some point, assume they decided to monitor pretty much all travel on the public roads: No one should travel anywhere unless independent review concludes that the risks are reasonable in relation to the potential benefits. Cf. Nat’l Bioethics Advisory Commission, I Ethical and Policy Issues in Research Involving Human Participants ii (August 2001). In other words, no one should be judging whether to go anywhere for him or herself, until after someone else has decided it’s okay. See C. K. Gunsalus, The Nanny State Meets the Inner Lawyer: Overregulating While Underprotecting Human Participants in Research, 14 Ethics & Behavior 369-382 (2004).

At least for now in this hypothetical world (FN1), travel plans on private roads are not covered at all, and many travel plans on public roads are exempt from the DMV’s oversight. If it involves travel on public roads, however, only the DMV can judge it as exempt. Some common plans are pre-approved as exempt, e.g., travel to local grocery stores along certain routes. But you probably still need to send an e-mail to your DMV notifying them of your intent to use these pre-approved plans. They’d like to know what you’re up to.

New Blog Forums

We have added two new blog forums to the schedule. The first is on the topic of Institutional Review Boards, a.k.a. Human Subjects. It’s scheduled for March 27-31, the week before the IRB conference at Northwestern University. The second is on the New Legal Realism Project and is scheduled for June 19-23.

The current participants are listed on the right. More names to come.

To Tell or Not to Tell

I appreciate Todd Peppers’ and Artemus Ward’s responses to my post. But I didn’t say (or mean) that "the journalists are spoiling it for the academics." Rather, I think that most law clerks make no distinction between journalists and academics when deciding whether and how to respond to requests for information. It’s not a distinction I’ve considered myself until now. And to be honest, I remain unpersuaded that it’s the right distinction to make. The fact that academics are less likely to achieve fame and fortune as a result of their work, for example, than are journalists or muckrakers does not persuade me that law clerks should talk to them more freely. Nor do I believe that it is necessarily true that "both clerks and justices have nothing to fear from us poor, dull, eggheads." Although I have not yet read Artemus’ forthcoming book (other than the introduction, which he very kindly shared with me last fall), the press accounts I have seen (those pesky journalists again!) suggest that it is critical of the role of law clerks and Justices’ dependence on them. It may or may not turn out to be a best-seller, but it also is unlikely to reassure those skittish research subjects that there are no downsides to participating in academic work.