Empirical Legal Studies

State Court Organization, 2004

A treasure-trove of data and information on our state court systems is contained in the recently-released State Court Organization, 2004.  Moreover, the Bureau of Justice Statistics (BJS) makes available much of the source-data in user-friendly formats.

According to the good folks at BJS, the publication "presents detailed comparative data by State trial and appellate courts in the United States. Topics covered include: the number of courts and judges; process for judicial selection; governance of court systems, including judicial funding, administration, staffing, and procedures; jury qualifications and verdict rules; and processing and sentencing procedures for criminal cases. Diagrams of court structure summarize the key features of each State's court organization."

Securities Law, PSLRA, and Legal Impact

Stephen Choi (NYU) and Bob Thompson (Vanderbilt) recently circulated Securities Litigation and its Lawyers: Changes During the First Decade After PSLRA which might hold particular appeal for those interested in "possible new lines of research into the effectiveness of the PSLRA."  An edited abstract follows:

"This Article surveys the empirical evidence on the impact of the PSLRA, examining the specific categories of reforms introduced by the Act.
                    ***
We report that while the market concentration of plaintiff law firms based on settlement amounts did not change appreciably after the enactment of the PSLRA, the tendency of top tier law firms to associate with lower tier firms did increase significantly in the post-PSLRA period.  We also report that institutional investors taking on the role of lead plaintiffs in the post-PSLRA period tended to develop repeat relationships with select top tier firms."

A View from the Ground. Part 2: Correcting Lawyer-Centricity

In order to understand what civil justice is and does, we need some information about what it doesn’t do and isn’t. My contention, a recurrent one in the literature, is that we rely too heavily on studies of formal institutions and their staffs as means of understanding how civil justice actually operates. I am not criticizing research on courts and lawyers per se -- indeed, I do work in this area myself -- which continues to be a valuable and important enterprise. But this research provides only part of the picture; an over-emphasis on lawyers and what they do results in a myopic, lawyer-centric view of what civil justice effectively is and actually does.

Qualitative research into disputing behavior and legal consciousness has offered us a broader view for many years. Survey research is beginning to catch up. A new generation of survey-based work returns to studying civil justice from the ground up, inquiring into normal people’s everyday experiences of potentially legal problems.

The new research improves on the old in a number of ways. I’ll pick just one:   

Database Links

If you scroll down the typelists on the right of this blog, you'll notice that I've added a new list entitled "Databases" (between "Conferences" and "Recent Posts").  This list provides links to a number of useful data sources such as the Songer U.S. Court of Appeals database, the Spaeth Supreme Court database, and Keith Poole's common space scores.   I hope our readers find this useful.  I'm now taking requests to add more online databases to the list.  Comments are open.

A View from the Ground. Part 1: Studying Justiciable Events

Justiciable events -- happenings and circumstances that “raise [civil] legal issues,” whether or not people “recogniz[e] them as being ‘legal’” and whether or not they respond to the event by using some part of the civil justice system (Genn 1999:12) -- provide a powerful analytic tool for understanding the workings of civil justice. Over the next week, I want to take the opportunity of this blog to introduce the study of justiciable events, to discuss some excellent, fascinating research from other countries, and to make a plea for American scholars to get involved.

Given the most recent available data, we can estimate that about half of the low- and moderate-income households in the United States are in the midst of experiencing at least one “justiciable event” (American Bar Association 1994: Table 3-1; Curran 1977). In a country with nearly 300 million people residing in more than 100 million households (Simmons and O’Neill 2001), that’s a sizable number of people and families exposed to problems with civil legal aspects. The most commonly experienced events involve money -- credit, bills, livelihood, benefit payments -- and housing-- landlord/tenant problems, mortgages and rent, repairs and property tax assessments -- but households also report family matters, personal injuries, civil liberties and civil rights violations and problems with children’s schooling, health care and municipal services.

 Justiciable events thus constitute a set of substantively diverse, frequently experienced, widespread events that provide researchers opportunities to observe people’s understanding of potentially “legal” problems, their interactions with staff from the civil legal system and with other third parties, and the consequences of different courses of action, as well as inaction. Older research understood these events as problems that constituted legal needs – i.e., as legal problems benefiting from a lawyer’s assistance. Following the life histories of justiciable events provides a view of civil justice “from the ground.” By conceptualizing such happenings as “justiciable,” the new generation of access to justice research [see the next blog posting] opens up the analytic space to explore the many ways in which people may think about and handle such events. This analytic space allows not only greater social scientific understanding, but also greater creativity in social policy.

The last large-scale, nationally representative social scientific surveys of Americans’ experience of justiciable events were the Civil Litigation Research Project (Trubek et al. 1983, archived at ICPSR) and Barbara Curran’s American Bar Foundation-affiliated national survey of household legal needs (Curran 1977). These were fine studies, but they are, at this point, going on 30 years old. The American Bar Association sponsored a more recent national survey, but even this is more than a dozen years old, represents only those households with annual incomes less than $60,000 in 1992, and was reported in highly aggregated tabulations meant to describe the volume and distribution of “unmet legal need” (American Bar Association 1994). If one wants more detail from this most recent survey, he or she is out of luck, because these data are not publicly released.

Stellar work is being conducted in other countries, including Australia, Canada, China, England,
Scotland and Wales -- about which more next time.

 

REFERENCES

American Bar Association. 1994. Report on the Legal Needs of the Low- and Moderate-Income Public. Chicago, IL:  American Bar Association.

Curran, Barbara A. 1977. The Legal Needs of the Public: Final Report of a National Survey.
Chicago, IL:  American Bar Foundation.

Genn, Hazel, with the National Centre for Social Research. 1999. Paths to Justice: What People Do and Think about Going to Law. Portland, OR: Hart Publishing.

Simmons, Tavia and Grace O’Neill. 2001. Households and Families: 2000. Census 2000 Brief. September 2001. http://www.census.gov/prod/2001pubs/c2kbr01-8.pdf.

Trubek, David M., Joel B. Grossman, William L.F. Felstiner, Herbert M. Kritzer, and Austin Sarat. 1983. Civil Litigation Research Project: Final Report. Madison, WI: University of Wisconsin Law
School.

 

Wall Street Journal Article on the Decline in Mass Tort and Class Action Lawsuits

The weekend edition of the Wall Street Journal (subscription required) has a interesting story entitled "Class Inaction: Plaintiffs' Lawsuits Against Companies Sharply Decline."   The alleged decline in mass tort and class action lawsuits does not come for a single empirical study.  Rather, the reporter cobbles together several disparate trends that ultimately support the story.

What really got my attention, however, was the comment by Geoffrey Miller (NYU Law) that "[t]he future of mass torts and class actions is very much in question."  Miller, who has authored numerous empirical studies on large scale litigation over last two decades (his more recent work posted here), is indisputably one of most qualified experts in the country to much such an claim.

Some of the trends cited in the story include:

• The indiction of Milberg Weiss (securities fraud filings are down 45% from 2005);
• A scathing 2005 opinion of U.S. District Court Judge Janis Jack that exposed rampant fraud in the silicosis, which has produced greater scrutiny by other courts.  (The story notes, "About 60% of the plaintiffs in the silica case had previously been plaintiffs in asbestos suits, even though it is extremely rare to get both silicosis and asbestos.  One doctor has made roughly 88,000 asbestos diagnoses over the years ... .")
• The Class Action Fairness Act of 2005, which diverts most class action cases into federal court, where the case law on class action certification is much tougher for plaintiffs.
• Numerous state tort law reforms,  such as medical criteria laws in Florida, Georgia, and Texas, which require doctor certification of asbestos or silica causation of illness in the early states of litigation, or punitive damage reform in Mississippi, which curtailed a previously attractive venue for plaintiffs' lawyers.
• Inspired by a rising tide of corporate friendly state and federal court rulings, Merck's willingness--and and success at--litigating Vioxx cases on a one-by-one basis, which has discouraged plaintiffs and lawyers from pursuing 60,000 of the estimated 100,000 potential Vioxx claims.

It is certainly true that some areas of class action and mass tort litigation have been tainted by massive abuse.  But it is also true that much of the recent class action and tort reform has been based on extremely shaky and exaggerated empirical claims.  See, e.g., Ted Eisenberg, Use it or Pretenders will Abuse It: The Importance of Archival Legal Information (short essay that discusses how major litigation reforms have often been at odds with the best available information).   The SSRN pages of Eisenberg and Miller, among others, have several studies that back this up.   

Over the last several decades, large scale plaintiffs' lawsuits have been a strong deterrent against dangerous or fraudulent conduct.  What, exactly, are the social costs of curtailing this litigation? Ultimately, legal empiricism is the key to striking the right balance. 

Guest Blogger: Rebecca Sandefur

This week's guest blogger is Rebecca Sandefur, Assistant Professor of Sociology at Stanford University.  Becky received her B.A. form the University of Wisconsin and her Ph.D from the University of Chicago.  Her research focuses on the intersection between the sociology of law and the study of inequality.  She co-authored Urban Lawyers: The New Social Structure of the Bar, and will be presenting a paper titled Lawyer, Nonlawyer and Pro Se Representation and Trial and Hearing Outcomes.

Lawyers are from Mars, Social Scientists are from Venus*

* With apologies to John Gray, and everyone else too.

Thanks to Jason and Co. for allowing me to guest-blog this week. I want to wrap up by talking about a couple of the things I see as an impediment to the ELS enterprise, and specifically to scholarly collaboration between legal academics and social scientists.

Traditionally, the American legal academy -- reflecting its primary mission to train professional advocates -- has generally taken what I think of as an adversarial approach to inquiry: One has a position to be supported, or a result to be achieved, and one marshalls evidence and arguments in support of that position/result while minimizing, discrediting, or otherwise undermining evidence and arguments that oppose it. The opposition does the same thing, and the best arguments (usually) win.

This model -- which, I think it's fair to say, pervades the legal academy -- gives rise to a number of externalities: an emphasis on rhetoric and argumentation, a prioritizing of deductive over inductive and abductive modes of reasoning, and an innate skepticism about facts and data. But while it is a tremendously powerful, efficient mechanism for accomplishing all kinds of things, it is a lousy model for "doing science;" and, in fact, is fundamentally at odds with how scientists think about what they do. (As I noted in a comment to an earlier post, I noticed this most starkly during my two years at NSF, when law faculty would often submit grant proposals that said, effectively, "If you fund my research, I will prove that...").

On the other hand, the intellectual culture of the social sciences brings its own pernicious effects to the collaborative relationship as well. The worst of these is a form of scientism I believe stems from social scientists' relatively low position in the scientific food chain. For many, it takes only a few instances of hearing that one's field "isn't a real science" to become unreasonably defensive and prone to elevate method over all else. One consequence of this is social scientists' tendency to denigrate legal academics' values, including the importance they place on individual cases (versus broad trends), doctrine (versus outcomes), and the normative implications of legal decisions (versus positive explanations of the decisions themselves).

My own view is that the best remedy for these meta-problems begins in the classroom. Law schools that offer courses in (social) scientific thinking -- and social science departments that welcome law school faculty, cross-list courses, and so forth -- underscore the idea that we share common pedagogical goals (critical thinking, logical consistency, etc.) and that both approaches are equally -- albeit differently -- valuable. Moreover, they provide a natural point of contact between the two cultures, and in so doing open up possibilities for greater cross-disciplinary translation and collaboration.

Metcalfe's Law

An interesting recent article in IEEE Spectrum has a nice discussion of how Metcalfe's law -- the maxim that the value of a network of size N grows quadratically (that is, proportionally to N²) -- is wrong, and why the actual growth rate is probably closer to N ln N. The intuition is that Metcalfe assigned equal value-added to all possible connections in the network, when in reality the marginal value of an additional connection between two nodes is decreasing in the number of possible connections. The paper suggests some interesting questions about the valuation of networks of various kinds, including those of interest to ELSers.

Generational Divide in Support for Environmental Policies

Viscusi and Hersch (Vanderbilt) have written The Generational Divide in Support for Environmental Policies: European Evidence.  The Abstract:

"This article examines age variations in support for environmental protection policies that affect climate change using a sample of over 14,000 respondents to a 1999 Eurobarometer survey. There is a steady decline with age in whether respondents are willing to incur higher gasoline prices to protect the environment. This relationship remains after controlling for socioeconomic characteristics. There are age-related differences in information about environmental risks, information sources about the environment, perceived health risks from climate change, and degree of worry about climate change. However, taking these factors into account does not eliminate the age variation in willingness to pay more for gasoline to protect the environment."

Conference on Empirical Legal Studies Preliminary Program

The First Annual Conference on Empirical Legal Studies to be held at the University of Texas School of Law (Oct. 27-28, 2006) has posted the preliminary conference schedule here.

They're From the Government...

...and they really are here to help you. I'm speaking about the National Science Foundation, and specifically about its Law and Social Sciences (LSS) Program. For the folks reading this, the LSS Program is probably the single best source for external funding of one's research; yet it's received relatively little exposure on this blog. While details about proposal submission deadlines, proposal formats, etc. can all be found on the Program's website, I want to highlight three common misconceptions about the LSS program that should be of particular interest to ELSBlog readers.

Misconception #1. The LSS Program (and/or the NSF in general) only funds quantitative research.

In fact, nothing could be further from the truth. Not only does LSS (and NSF) fund qualitative research (for examples, take a look at some of the recent awards made by the program), but the Program and the Foundation have been active in helping to developing standards for evaluating qualitative research, and in strengthening qualitative methods in the social sciences.

Misconception #2. There's not enough money there to bother going after.

The last time I checked, the LSS program was awarding grants totalling around $3.5 million a year. That's not crazy money, but it is a substantial amount nonetheless. And, as I often reminded researchers when I oversaw the Program, somebody's going to get it.

Misconception #3. It is improper/inappropriate to discuss my project/proposal with the NSF program officer.

Again, nothing could be further from the truth. NSF program officers (POs) are the individuals who oversee the proposal, review, and award process; their job is to serve their program's research community. As long as you don't try to bribe them, threaten them, or otherwise violate federal law, they are generally happy to answer questions about the proposal process, review criteria, and other aspects of the funding game, and even to offer suggestions. In fact, a short discussion with a program's director is probably the best place to start a proposal, since POs can often tell you immediately whether or not your idea is a good "fit" for their program (and suggest an alternative if it is not). (FYI, the current LSS Program's officers are Isaac Unah and Kevin Gotham).

The Polls - Name Two Justices

I got the full report from Zogby on the recent pop culture survey. In this survey, people were asked to name only two justices, not all nine. Sixty-one percent of people could not name any justices, whereas 15% named one and 24% named two. The press release characterizes Thomas as the “most recognized justice,” and while prior surveys support this characterization (now that O’Connor is gone), the Zogby survey on its own probably supports calling him only the “most top-of-mind justice.”

Here are the percentages indicating how often each justice was named:

Thomas	20%
Scalia	11%
Roberts	9%
Ginsburg	9%
Alito	5%
Kennedy	4%
Souter	3%
Stevens	2%
Breyer	1%

Another relevant question was asking people to name the three branches of government. Forty-two percent named all three, 17% gave an incorrect “other” answer, and 41% were “not sure.” Unfortunately, it’s not clear what exactly counted as a correct answer to this question, e.g., did “President” count for the “executive” or the “courts”  for the “judiciary”? Without this information, this question is not as interesting.

n = 1213
error: +/- 2.9%

Source: Zogby International, “Pop Culture/Political Survey: Nationwide and in Ten Metropolitan Areas” (Aug. 10, 2006).

WPSA - Call for Papers

If Chicago and the MPSA are not your thing, then maybe Vegas and the Western Political Science Association annual meeting will better suit your fancy. It all happens March 8-10, 2007 at the La Riviera Hotel. The deadline for papers is coming up fast, September 11, 2006. Go here for all the details.

MPSA - Call for Papers

The 65th annual meeting of the Midwest Political Science Association is scheduled for April 12-15, 2007 at the Palmer House Hilton, and the deadline for proposals is October 2, 2006. More information can be found here.

"Data" → "Agenda"?

At the end of an earlier post, my friend and former boss Rick Lempert touched on a pet peeve of mine: treating "data" as a singular noun (as in, "Our data is...," or "The data suggests..."). "Data," I learned early on in my graduate training, is the plural of "datum," and so is akin to "facts"; the only correct form is therefore the plural ("Our data are...," "The data suggest..."). Any other usage -- including that in a number of posts on this very blog -- is simply wrong.

Sadly, my smug self-satisfaction at relentlessly educating all those singular-data cretins has come to an end. As it happens, the cretins were right; or, at least, they weren't as wrong as I'd thought. Language, as we all know, evolves; the growing consensus is that data is a mass noun, and so admits to a singular usage. In this respect, data is undergoing the same transition that agenda -- formally, the plural of agendum -- underwent a long time ago: Few people I know would say (e.g.) "The meeting's agenda allocate ten minutes for opening statements." My reading of a number of discussions of this issue around the blogosphere also suggests that there is a generational component to the change, with younger folks being more willing to accept -- and in some cases, even argue strenuously for -- data as singular.

So, with a heavy heart, I'll be scratching data-as-singular-noun from my pet-peeve list (which includes, among other things, reporting "p = 0.000" and stating that one's parameter estimates are "significant in the wrong direction"). But with that, a plea: keep treating data as plural anyway. It's still correct, and (in this blogger's opinion) it just sounds better.

Lott v. Levitt: Shermer in Scientific American

Michael Shermer comments on the case in the September issue of Scientific American. His conclusion: "Mr. Lott, tear down this legal wall and let us return to doing science without lawyers. Replicating results means testing hypotheses--not merely duplicating methodologies--and this central tenet of science can only flourish in an atmosphere of open peer review."

(via Tim Lambert)

Peer Review Redux

I appreciate Jason's invitation to guest-blog. I'll try to resist the urge to promote my recent paper (with Todd Peppers) on Supreme Court law clerks, and instead focus my comments (mostly) on the ways in which the cultures of the legal and social science academies are sometimes in opposition, and if (and how) those tensions might be resolved. I'd like to start with the issue of peer review, a subject that has received a good bit of attention both on this blog (e.g., here, here, and here), and on several others.

The advantages of peer review (by which I mean single- or double-blind review of singly-submitted manuscripts, as is the standard practice in the sciences, including the social sciences) are well-understood: it ensures (or at least improves the chances) that reviewers will have the necessary expertise to evaluate the work, allows those reviewers to be frank in their evaluations without fear of retribution, and removes at least some of the potential for reviewer bias due to personal factors. While the system generally works well, it is also criticized for (among other things) being slow, inefficient, biased in favor of well-known scholars and/or institutions, subject to predation and plagiarism, and inherently conservative.

All of this is in stark contrast to law reviews, where multiple submissions, expediting, and student review are the norm. Law review turnaround is generally faster than that at refereed journals, and its proponents claim that it lacks the conservative bias that comes from older, more established established scholars acting as gatekeepers. At the same time, it is hard for me to imagine a system more ill-suited to evaluating the scientific merit of a piece of scholarship. In a nutshell, peer review's small-"c" conservative bias is good for the scientific enterprise, but sometimes comes at a cost to the publishability of really innovative work, while law journals' biases toward "hot topics" and unconventional ideas works in the opposite direction.

The implications, particularly for junior faculty, are clear. Social science departments devalue publications in non-peer reviewed outlets, leaving social scientists with an interest in the law little incentive to publish in even the most prestigious law journals. On the other side, law faculty with shorter tenure clocks are less willing to allow 2-3 years (or more) to pass before an empirical paper is published in a peer-reviewed journal, making law reviews all the more appealing. The overall effect is to exacerbate what Frank Cross has called the "unfortunate interdisciplinary ignorance" between legal academics and social scientists.

What to do, then? One alternative, suggested by Valerie Hans in a comment to an earlier post, is to write separate pieces, one for a law review audience, the other for peer-reviewed journals. More radical options call for altering the way the journals (particularly law journals) operate. One suggestion along these lines was offered a few years ago by Lee Epstein and Gary King, but many others exist. One I find particularly creative (and intriguing) is open peer review. The journal Nature has recently implemented this approach on a trial basis; their report on the experiment is due out this fall. Given law professors' apparent affinity for on-line scholarship (cf. SSRN, and this blog!), this strikes me as a particularly strong candidate as a model for peer review of empirical legal research.

Guest Blogger: Christopher Zorn

This week's guest blogger is Professor Christopher Zorn.  Chris' research interests are in the areas of law and judicial politics, quantitative methodology, and international politics.  His current research examines a range of topics in American and international political institutions, including federal government litigation, U.S. Supreme Court decision-making, and international relations, as well as research on event count, survival, and panel data models for quantitative research.   He is formerly a Visiting Scientist and Program Director for the Law and Social Science Program at the National Science Foundation.

More on Global Cities and Labor Market for Lawyers

My fascination with Global Cities continues.  As I noted in an earlier post, Global Cities are centers of international commerce that are defined by concentrations of advanced producers services--i.e., bankers, accountants, consultants, advertising executives, and, of course, lawyers.  As it turns out, a globalizing ecomomy has two geographic effects that move in opposite directions : (1) the decentralization of production, as companies piece together the most cost-effective way (often through elaborate subcontracting) to make and distribute their goods; (2) a  geographic centralization of corporate strategy functions to control more sophisticated global operations.   

Because of the complexity, scale, and stakes of most global ventures, hiring outside counsel is increasingly a matter of expertise (and value) rather than per-hour billing cost.  This expertise can only be acquired through exposure to a large volume of high-end matters.  Hence, a disproportionate amount of the most sophisticated U.S. legal work is performed in a handful of global cities.  And to access this human capital, firms (and ultimately clients) are willing to absorb labor and real estate costs that rapidly outpace inflation.   Although many Fortune 500 have exited these same high cost markets, agglomeration provides significant synergies for advanced producer services. 

Here is a graphic that summarizes the Top 10 U.S. markets in terms of growth of corporate lawyers.  The MSAs in green are global cities under virtually all the competing methodologies; the  other five metropolitan areas are on the cusp. [click on chart to enlarge]

Below the fold are additional graphics that show how these numbers appear to be trickling down to law students evaluating their future employment prospects.

Thanks!

I thought I might have time for one more post focusing on some of my ongoing empirical projects, but alas, the duties of the fall semester are upon us and my time here at ELS Blog must come to an end.   I want to thank Jason, Bill and the rest of the ELS bloggers for having me as a guest, as well as those who took the time to comment on my posts or send me e-mail responses.   ELS Blog has quickly become one of my favorite places on the web and I look forward to reading your continued efforts to bring "methods to our madness."  Of course,  Truth on the Market is my most favorite place on the web, and the place to which I shall now return for regularly scheduled blogging activity.  I invite ELS Blog readers to stop by and check us out anytime.

Berman on Blogs ... and an Interesting Data Point

This post by Peter Lattman at the WSJ Law Blog caught my eye.  It is a short review of  Doug Berman's (OSU Law) "Scholarship in Action: The Power, Possibilities, and Pitfalls for Law Professor Blogs,"  which was originally presented  as part of the symposium on legal blogging sponsored by the Berkman Center for Internet & Society at Harvard Law School.   

Berman, who edits the Sentencing Law & Policy Blog, does not need to be remotely defensive on the relevance of blogging to scholarship.   His blog, which has garnered over 20 citations in judicial decisions (as of April 2006), has been endorsed by the President of the Federal Judges Associations and cited in testimony before the Senate Judiciary Committee (see this WSJ online story).  Berman's essay is a thoughtful reflection on the purposes of legal scholarship--which, Berman observes,  has become remarkably undertheorized due to a process of institutional drift over the last few decades.    Berman believes that blogging is a medium that provides large synergies between the three traditional dimensions of a law professor's job: teaching, scholarship, and service.  Thus, Berman concludes that blogging is destine to have an impact and may someday be "seen--along with 'serious legal scholarship'--as a preferred form of law professor activity."  I agree.

The reason that the Berman post caught my eye (and why I downloaded the paper) was a reference to statistics on the average length and number of footnotes in the Harvard Law Review in 1950, 1975, and 2000.  The following chart, from page 3 of Berman's paper, speaks for itself [click on chart to enlarge].

The Polls - Another Name-the-Justices Survey

Zogby released (some of) the results from a new pop culture knowledge poll yesterday. It included a couple of Supreme Court items:

When asked to name two of Snow White's Seven Dwarfs and two of the nine U.S. Supreme Court Justices, 77% of Americans polled were able to identify two dwarfs, while only 24% could name two Supreme Court Justices. Not surprisingly, Clarence Thomas, whose nomination was marked by controversy, was the most recognized Justice -- identified twice as often as his next best-known peer on the Supreme Court -- Antonin Scalia.

And the other one:

Surprisingly, despite record-breaking ratings, when asked who the newest American Idol is as of May 2006, only 23% of those polled named Taylor Hicks. Still, while that may seem low, it was double the percentage of those who recalled which Supreme Court Judge was confirmed in January 2006. Only 11% answered "Samuel Alito."

As best I can tell, Zogby has not posted the full results of this survey, but the press release is here. It contains additional results that may be interesting but that are not really ELS relevant, e.g., "Krypton was identified correctly by 60% of the respondents as Superman's planet of origin, while only 37% were able to name Mercury as the closest planet to the sun." (When the new movie came out a few months ago, I actually discussed this very question of how many people would be able to name Krypton as Superman's home planet with a friend of mine. Now, thankfully, there are data to resolve this question.)

My collection of previous name-the-justices poll results is here. I see that I still need to update it with a fourth poll from 1989; I had forgotten about it when I did the original post.

UPDATE: William Child at TortsProf points to a very nice analysis at the Language Log of how these public knowledge surveys can be spun by those who write press releases. A second Language Log post on the subject can be found here. Both posts focus on the First Amendment survey sponsored by the McCormick Tribune Freedom Museum.

Choosing Law, Forums, and Delaware Paper

My Cornell colleague (Ted Eisenberg) and former Civil Procedure teacher (Geoffrey Miller) recently circulated an interesting paper (Ex Ante Choices of Law and Forum: An Empirical Analysis of Corporate Merger Agreements ) arguing, in part, that "Choosing Delaware as the incorporation situs is, effectively, a choice-of-law decision."  Excerpts from the abstract follow:

Legal scholars devote much attention to the incorporation puzzle - why corporations so frequently incorporate in Delaware. This paper suggests that focusing on the incorporation decision overlooks a broader but intimately related set of questions. Choosing Delaware as the incorporation situs is, effectively, a choice-of-law decision.

                                                                          ***
We study a data set of 412 merger and acquisition contracts contained as exhibits in SEC Form 8-K filings over a seven month period in 2002 to assess the decisions the parties have made regarding choice-of-law and choice-of-forum. Although these contracts frequently select Delaware law and Delaware as a forum, there is a relative "flight" from Delaware in this contractual setting. Delaware corporations choose Delaware law less than other corporations choose the law of their state of incorporation. Furthermore, many contracts specifying Delaware law did not specify Delaware as the litigation forum.

George J. Stigler: The Father of Empirical Law and Economics?

Ronald Coase is conventionally described at the "Father of Law and Economics," sometimes along with Aaron Director, Gary Becker, and Guido Calabresi.  Rightly so.  Coase's The Problem of Social Cost  is of primary importance to law and economics by highlighting the importance of transactions costs for understanding law as an economic system.  Calabresi's contributions follow in this spirit.  Others might be considered "founders" of various branches of law and economics, e.g. Douglass North and New Institutional Economics, Gary Becker and Price Theory and the Law. 

I understand there is an obvious superficiality involved when one purports to make claims about the  "founders" of any "movement," but don't want to get bogged down in questions about what qualifies one as a "founder" (e.g., didn't Adam Smith found empirical law and economics?) nor the definition of a "movement" of scholarly thought.   My simple point is that empirical law and economics seems to have developed its own identity in legal scholarship separate from economic analysis of the law dedicated to Coasean transactions cost analysis or Beckerian price theory, though each of the latter certainly include empirical components.  To the extent that it is useful to associate a specific methodology with a scholarly movement, and I think it is for reasons I will discuss, I humbly suggest that George Stigler's research agenda is an excellent model for empirical law and economics scholarship, if not empirical legal scholarship more generally.  I offer some support for this claim below the fold.

Is Antitrust An ELS Success Story?

First, I would like to thank Jason Czarnezki for inviting me to contribute to the ELS space as a guest-blogger this week. Following up the excellent series of posts by Richard Lempert on the ELS movement is no easy task.  I have neither sufficient experience as a practicing social scientist nor in the legal academy to supply any meaningful insights on the ELS movement's placement in historical context.  But I do hope to start with some thoughts about what the contributions of empiricism to antitrust might tell us about both the promise and limits of ELS (can I substitute empirical law and economics and ELS interchangeably in this context without loss of generality and without offending anybody?).

After all, antitrust represents a best case scenario in terms of susceptibility to empirical influence. The conditions for impact are ideal. Professor Lempert’s last post noted that antitrust was somewhat immune to the legal academy’s “relative inattention to social science.” Why? Well, let me start with what I hope to be two relatively non-controversial and highly correlated assertions. The first is that antitrust is uniquely suited to benefit from economic analysis, especially antitrust empiricism. The second is that antitrust doctrine has embraced economic analysis and empiricism. This was not always so. But there is no real dispute that antitrust welfare economics has won the day. Modern antitrust doctrine directly incorporates economic analysis: will a merger increase prices? Will the conduct at issue produce an anticompetitive effect? What is the impact of the conduct on consumer welfare?

What then, does a body of law that expressly incorporates economic thinking, and particularly empirical learning about business practices, look like? Are there lessons to be learned for other areas of law attuned to empirical learning or is antitrust so uniquely situated that the ELS/antitrust experience should be disregarded for this purpose? While I don’t have conclusive answers to all of these questions, I suspect that the successes and failures of empiricsm and antitrust over time can tell us something about both the potential and limits of ELS more generally.   

Lott v. Levitt: Coverage in Chicago Magazine

James Meriner has an article on the relationship between Lott and Levitt in the August issue of Chicago Magazine.

Via Tim Lambert

Guest Blogger: Josh Wright

Josh Wright, an Assistant Professor of Law at George Mason, is a graduate of the UCLA School of Law, where he was a managing editor of the UCLA Law Review. He also holds M.A. and Ph.D. degrees in Economics from UCLA.

Josh's research focuses on the law and economics of the competitive process for product distribution, including slotting allowances, category management, payola, and other marketing relationships.  Josh is also a regular contributor to Truth on the Market, a weblog dedicated to academic commentary on law, business, and economics.

The Polls - Occupational Trust

The Harris Poll has some new data on the most and least trusted occupations. Respondents were asked, "Would you generally trust each of the following types of people to tell the truth, or not?"

Doctors are at the top with 85% percent generally trusting them. Actors are at the bottom with only 26%. A few other categories:

Teachers: 83%
Professors: 75%
Judges: 70%
The President: 48%
Members of Congress: 35%
Lawyers: 27%

Additional information, including the results for all 22 occupational categories, can be found here. Unfortunately, the results do not reveal what happens when you combine a professor and a lawyer.

Data in Environmental Law

As a scholar who focuses on empirical legal studies and environmental/natural resources law (and hopes that the two areas will "talk" to each other more often), long ago I wrote about the need for better data in crafting environmental and natural resources policy.  Holly Doremus (UC-Davis), in her paper Gaps in Natural Resource Management: Sniffing for Leaks Along the Information Pipeline, has offered some suggestions for improving the information and data pipeline in natural resource policy.  Abstract below the fold.

ELS Past, Present and Future

In the 1930s, Yale Law Professor Underhill Moore, enlisting the young Lloyd Cutler as a research assistant (along with all of Cutler’s classmates – they had no choice), did a massive empirical study of how New Haven’s citizens responded to the presence of parking meters and sanctions for not feeding them. The resulting article, whose length was exceeded only by its tedium, may have doomed empirical legal studies for the next 20 years. Yet in many ways the project was well conceived. Parking behavior is a nice way of studying how ordinary citizens respond to mild sanctions. Parking behavior is easily observable, as are the costs of parking and the cost and likelihood of sanctions. Various independent variables, such as the value of a car (as an index of wealth) and the gender, race and approximate age of the driver may also be captured with considerable reliability, and enough cases may be generated to allow detailed analysis. But Moore lacked the technology to adequately analyze the data and to present his findings in ways that could be easily understood. This was more or less true of quantitative research on law, and most other areas, until the 1960s.

Outside of economics and psychology, the classic empirical social science offerings before that decade were mainly qualitative studies, books like William Foote Whyte’s Street Corner Society and Philip Selznick’s TVA and the Grass Roots, though a decade earlier, with the leadership of people like Paul Lazardsfeld at Columbia and the survey researcher Angus Campbell at Michigan and their students, modern quantitative social science was being born. By 1970, in all the social sciences except anthropology, quantitative research and formal modeling had replaced qualitative studies and informal theorizing as the hottest, and in some departments the only respectable, way to do social science. Yet except for a few celebrated works, like Kalven and Zeisel’s The American Jury and Wayne LaFave’s massive study of arrest processes, works that used numbers crudely if at all, legal scholarship was barely affected.  Underhill Moore’s ghost had not risen.

I think there were many reasons for this, including a law school culture that reserved its highest accolades for those most adept at analyzing legal doctrine, particularly the pronouncements of the Supreme Court; the view that those who had excelled academically at top law schools and clerked for the right judges needed no further training beyond, perhaps, a bit of practice experience before joining the nation’s leading law faculties, and, in many of the nation’s law schools, a two-and-a-half year, one grand article, path to tenure that did not provide the time needed for empirical data collection and analysis and did not recognize that good quantitative social science more often consists of the production of a series of fine bricks than the construction of one imposing wall.

Some Simple Cautions - Day 4

 Central to most empirical research is the control of plausible rival hypotheses. If we are trying to show empirically that the world is or operates in one way, we need to convince the reader that our results are not likely to be attributable to explanations that undercut or are inconsistent with the description or explanation we provide. The key concept here is plausible. We need not dispose of all possible explanations for our results. If we had to do this, we never could publish. But some law professors, many lawyers, and too often courts, are adept at ignoring the plausibility constraint when they want to reject an empirically-based argument. Instead they treat, and get away with treating, possible alternative explanations as if their mere possibility seriously undercuts reliable data and well done analysis.

 David Baldus and his coauthors, if I recall correctly, included more than 600 independent variables in a model aimed at investigating whether death sentences in Georgia were tainted by race effects. The goal, I assume, was to fend off this kind of uninformed (or bad faith) prosecutorial and judicial nit picking. From a scientific standpoint, the far more parsimonious model they included, with 32 variables as I recall, was a better way of examining the data. But their tactic worked – in a way. When the McCleskey case, in which the Baldus study was the key evidence, reached the Supreme Court, the conservative majority chose not to contend with the analysis, but instead held that in order to show the racial discrimination needed to reverse a death sentence a defendant had to show that in his particular case discrimination played a role in his sentence.  This meant the Court never had to confront  the Baldus finding of substantial race of victim discrimination when cases were of intermediate seriousness.  It had become irrelevant.

Theory, Context and Common Sense

Day – 3

 I would like to talk to today about three aids to assessing empirical work: theory, common sense and knowledge of context. They are equally valuable for those doing the work.

 In 1975 the economist Isaac Ehrlich published a time series study of the deterrent effects of capital punishment running from the late 1930s to the mid 1960s, using what were at the time advanced techniques of statistical modeling. His results, which indicated that one execution deterred 7 or 8 homicides, were attention grabbing to say the least, particularly since they were set against a background of numbers of earlier studies, using apparently simpler methods, almost all of which found no evidence that capital punishment deterred homicides, and some of which suggested a possible brutalization effect. Ehrlich’s findings gained credibility by the apparent statistical sophistication of his methods (with tongue only slightly in cheek, I would define statistically sophisticated methods as methods with lots of equations and numbers that many readers cannot or do not have the time to follow).

 Because the topic was so controversial, other social scientists reexamined Ehrlich’s data using equally sophisticated, and often similar, statistical approaches. These reanalyses revealed that Ehrlich’s model and his results were acutely sensitive to the years analyzed. If the last years of the time series – the 1960s – were removed from the analysis, evidence for a deterrent effect disappeared, and in some studies a brutalization effect seemed possible. The 1960s were, of course, a period when executions for murder dropped to almost zero and all sorts of crimes were rapidly increasing. This replication research pretty much destroyed the credibility of Ehrlich’s results in the academic community, though their influence on the popular debate, like the Cheshire cat’s grin, long lingered.

Day 2 _ What's in a Name?

     I might as well say it at the outset; I don't like the nomenclature adopted by the ELS movement.  I don't like "Empirical Legal Studies" as a name for a type (dare I say "school") of scholarship, and I positively bristle when people are called by their colleagues or describe themselves as "empiricists" or "legal empiricists."  One reason for this, I am sure, is that where I (in part) come from - Sociology - to call someone an empiricist - or, even worse, a "dust-bowl empiricist" was not flattering.   It implied a person who crunched numbers with no concern for theory and often without an inkling of what the numbers represented.  Relationships are often easy to spot in data, but understanding what they mean and, in particular, the mechanisms that underlie and produce them is unlikely to result from number crunching alone.   I wish I could say that legal empiricists have gotten beyond this, but too often I feel that the power of the number crunching that one now sees in law reviews is not matched by an equally sophisticated sense of how plausible theories are built and tested and how causal arguments are structured. 

    This is not to say that there is not considerable value in organizing complex data sets and laying our legally-relevant relationships.  There is, and people who do this in an unbiased methodologically sophisticated manner make a genuine contribution.  But if the goal is not simply to see what is out there, but to understand processes and institutions, too often empirical scholarship  in law (though not only in law) not only falls short of the mark but overstates what analysis reveals.

     But there is another more fundamental reason why I don't like the idea of defining some portion of the legal scholarly establishment - namely those doing quantitative data analysis - as empiricists.  It seems to me that this ignores or denies the way in which legal scholarship, as traditionally done, is the most empirical of professions.  The most traditional lawyers work with empirical data all the time - case texts.  Texts are out there as part of the real world, and from the point of view of empirical scholarship have some substantial virtues.  In particular they are transparent (even when opaque), equally accessible to all scholars and open to various methods of interpretation, some of which are better than others.  They also often matter.  Would that the data "legal empiricists" examined always had these virtues.

      There is also the question of qualitative data.   I am distant enough from the ELS movement that I do not know how its core advocates regard qualitative research, but taking down 5 volumes of the Journal of Empirical Legal Studies that happen to be close at hand I could not help but note that every article in every volume had a quantitative dimension.  Each had at least a graph, table, equation or regression and most analyzed and presented results using more than one of these analytic modality.  Yet qualitative research is as empirically-based as quantitative research and it can be as unbiased and as rigorous.  Moreover, it is often more revealing of relationships legal scholars seek to understand, not to mention more accessible and interesting.    Lawyers have done many quantitative studies I find useful and admire, but I would not elevate any of them above, for example, Bob Ellickson's study of Shasta county when it comes to developing and sharing an understanding of the real world or, in this case, illuminating the limitations of the Coase Theorem. 

   New tools allow for the more systematic analysis of qualitative data and for sharing qualitative data in ways never before possible.  A cutting edge in social science research, including such heavily quantitative specialties as survey research, is the integration of qualitative and quantitative methodologies.   Should the law school world fission into communities that exacerbate divisions between qualitative and quantitative researchers and between those interested in numbers and those analyzing doctrine, I think we will be falling behind the times in social science, and our understanding of the law and legal systems will be the worse for it.

     I am not saying this is happening or that the movement to create "empirical legal scholarship" is pushing us in this direction.  Indeed, the best people doing such scholarship are often the most interested in what non-quantitative research has to offer.  But I do wish the law school world would find a different term than "empiricist" to describe people who work more with quantitative than with qualitative or doctrinal data.  In the law school world, we are, almost all, empiricists.   

   Rick

Tomorrow:  Common Sense and Theory

Reflections on Empirical Legal Studies

    I would like to thank Jason Czarnezki for inviting me to be this week's guest blogger on the Empirical Legal Studies web site.  Readers will find also an extended comment I made in response to a comment by Bill Henderson on Richard Sander's research, but this is not the intended topic of my blogging (Though I may return to it at the end of the week.)  Rather I want to use my space to simply to reflect on the Empirical Legal Studies movement and to suggest some cautions for those doing empirical research on legal topics. 

       I have been doing empirical work on legal issues as a law professor for almost 40 years, long before anyone spoke of empirical legal scholarship and legal empiricists.  I have been making the case for the virtues of empirical scholarship on law for almost that long.  Thus I am delighted to see the expansion of empirical legal studies (within and) beyond economics, and to know that there is a sufficiently large community of scholars who identify with the area as to justify a journal and a web forum, not to mention an AALS Meeting theme.  At the same time I have concerns about the quality of some of the empirical work being done and how empirical work, even good scholarship, is used.  To put it bluntly too much empirical scholarship is being deployed normatively, downplaying caveats that should be attached to findings, and some research, including work by outstanding scholars, seems strongly agenda driven (sometimes to the point of being financed by parties building records for litigation).  I also see in some work a divorce of empirical analysis from theory and context which not only can diminish the utility of empirical studies in building more general understandings, but can also lead to poor research designs and misunderstandings of data.  Moreover, even when research is of high quality and done with great care, its results can be hijacked by groups that oversimplify what was found in order to  "sell" positions they hold and would hold even if the empirical work had come out differently.   

     Too often researchers encourage misuses of their results in conclusions that push the practical implications of their research, even when the more detailed analysis emphasizes proper cautions.  While this occurs with empirical students of the law in liberal arts schools by political scientists, sociologists, economists and psychologists among others, the problem tends to be more severe in the empirical work of law professors, perhaps because most see their business not as building social or behavioral theory but as criticizing laws and legal institutions and recommending reform.

Lempert Comment on Affirmative Action Post

A couple of weeks ago I posted a lengthy essay on the debate surrounding Richard Sander's affirmative action research.   Richard Lempert, our guest blogger for the coming week and one of the most prominent Sander critics, posted a detailed comment to my post, which I encourage all our regulars to read.   The comment is here.

Richard, we look forward to your posts.  Welcome.  bh.

Guest Blogger: Richard Lempert

This week's guest blogger is Richard Lempert of the Michigan Law School.  Professor Lempert is particularly concerned with the problem of applying social science research to legal issues. This is reflected in much of his work, particularly his work on juries, on capital punishment and on the use of statistical and social science evidence by courts, as well as in his service as an original panelist in the National Science Foundation's Law and Social Science Program and with the National Research Council's Committee on Law Enforcement and the Administration of Justice, which he chaired from 1987-89. He is the author (with Joseph Sanders) of An Invitation to Law and Social Science and co-editor (with Jacques Normand and Charles O'Brien) of Under the Influence? Drugs and the American Work Force. From 1982-85, he edited the Law & Society Review. He is a recipient of the Law & Society Association's Harry Kalven Jr. Prize prize for outstanding socio-legal scholarship and has held visiting fellowships at the Center for Advanced Study in the Behavioral Sciences and at the Russell Sage Foundation. In 1993 he was elected a fellow of the American Academy of Arts and Sciences. He currently serves as the secretary of Section K of the American Association for the Advancement of Science.

Video Game Violence in Court

On Monday, the United States District Court for the District of Minnesota struck down the Minnesota Restricted Video Games Act. See Entertainment Software Assoc. v. Hatch, No. 06-CV-2268 (D. Minn. 2006). The Act prohibited minors [below 17] from knowingly renting or purchasing a video game carrying an ESRB rating of AO (Adults Only) or M (Mature). As of this moment, only 23 titles carry an AO rating (including only one game of any note, Grand Theft Auto: San Andreas), but 989 titles carry an M rating (including my own current favorite, Splinter Cell: Chaos Theory). The penalty imposed on a minor for violating the Act was a $25 fine. I'll focus on the use of the empirical evidence in this case.

For the Minnesota law to have survived under Eighth Circuit precedent, the state needed to “come forward with empirical support for its belief that ‘violent’ video games cause psychological harm to minors.” Interactive Digital Software Assoc. v. St. Louis County, 329 F.3d 954, 959 (8th Cir. 2003). It needed “substantial supporting evidence of harm.” Id. (internal quotation marks omitted).

The state relied primarily on a meta-analysis by Professor Craig Anderson of Iowa State University, who summarized his findings as follows:

Variable Transformation and Correct Parameter Estimations

Over the last few weeks, Andy Morriss and I have been working with U.S. News rankings, focusing primarily on post-graduation outcomes such as employment upon graduation and bar passage.   Much of the raw data does not satisfy the assumptions of ordinary least squares regression (linear relationship between X and Y, constancy of variance, etc.).  Thus, to create a proper linear model, I have been experimenting with various tranformation methods.   

One of the lessons this experience drove home was the importance of examining your data BEFORE you rely upon the various OLS parameter estimates.   In our case, the estimates changed in important ways after transforming the relevant variables.   It also reduced the amount of heteroskedasticity (i.e., nonrandom distribution of errors) in our various specifications, which will save us some additional corrective steps.

Here are some graphics that I thought readers might find useful.  This graph [Click to enlarge, excerpted from Fox, Applied Regression Analysis, Linear Models, and Related Methods 70 (1997)] shows a monotone curvalinear relationship, which can plausibly be corrected by a power transformation of the dependent (Y) or independent (X) variable:

"End of Term Statistics and Analysis"

Time to trek down to the Law Library... The recent issue of Preview of United States Supreme Court Cases has an article titled End of Term Statistics and Analysis (of the 2005 Term) by Thomas Goldstein.  I can't find the online version of the article, though the dagger footnote says it's available on SCOTUSblog.  The Harvard Law Review also annually provides SCOTUS Stats, usually in the November issue (see, for example, the data on the 2004 Term).

"My Professor..."

The two unrelated topics of "where to publish" and student course evaluations are relevant to two articles found in this month's P.S.: Political Science & Politics.   In Silencing the Dummy Variable: A Plea to Heed One's Audience and Publish More, Stephen Wasby suggests that in some circumstances it is appropriate to "tone down" the stats talk in articles, and perhaps write different versions of the same article to place in journals with different audiences.

In My Professor is a Partisan Hack: How Perceptions of a Professor's Political Views Affect Student Course Evaluations (click on title to download the pdf file), Kelly-Woessner and Woessner find that liberal professors are viewed "more favorably" and "[s]tudents who perceive their professors to be political allies rate courses more favorably than do students who perceive their professors to be political foes."

They go on:  "Consequently, if the goal were simply to win the love and adoration of the students, clever instructors would merely pander to the median 'voter.'  By mimicking students' views and reinforcing long-held beliefs, professors might score well on student evaluations, while providing no useful information at all."  The authors do not advocate this, but I think it points to some flaws in student course evaluations--what do they measure?; can they be manipulated?; is there a gap between student perception of what is good teaching and what is actually good teaching?

Grand finale

I think we are drawing to a close on our blog forum, and I think Jeffrey and Bill have graciously agreed to let me have the last word.  I greatly appreciate all the points and comments, and am (naturally) particularly fond of Bill's robust defense of teaching evaluations. 

As for the question of the complexity of the relationship I am actually planning to use the underlying data for two additional projects that may reach some of Jeffrey's concerns.  The first is a large scale regression analysis where I greatly increase the number of variables to include any publicly available AALS information on the law professors studied (such as gender, years in teaching, years in practice before teaching, law school attended, primary teaching area, etc.) to see if any of those variables correlate with teaching or research.

The second is a qualitative study of the professors.  I will start with a mailing inquiring about their own views on the interaction between teaching and research, and then probably follow up with some phone interviews.  With these additional data and observations I will hopefully be able to offer more than just the raw numbers.  Thanks again for the forum, and please don't hesitate to email any further feedback or suggestions.   Ben Barton.

Teaching - Writing Correlation

There is a lot of research on student evaluations, perhaps two-thousand articles. One meta analysis indicates that student ratings are affected much more by expressiveness than by content, while student achievement is affected more by content than by expressiveness.  See 'Educational Seduction', by Abrami, Leventhal, and Perry, in Review of Educational Research. That does not mean there is no correlation between ratings and effectiveness, but it does mean we need to be cautious in making conclusions about perhaps subtle effects on effectiveness when our data are from ratings. I am not criticizing Benjamin for using ratings; indeed, I admire and applaud his effort. I would just prefer that the conclusions be stated without the logical leap. What is wrong with saying, "I found no correlation between writing output and student ratings." That would let the reader decide how much to conclude about teaching effectiveness.

As for Bill's market point, I am not arguing that these educated consumers we call students do not know what they want or that they do not know it when they see it.  But I have little faith that they want what we educators think they need, as opposed to entertainment or bar passage. They could rationally choose to give up some small bit of lawyering effectiveness in the future for some entertainment now. I have even less faith that they want to learn what society needs them to learn for the good of all. I'd bet big money that I could raise my student ratings and lower the quality of my Property instruction by replacing future interests with more on landlord-tenant law.

The more important point that I wanted to make is that Benjamin seems to be looking for a simple relationship when a complex one is more likely. To carry my optimization point further, suppose that most faculty members have experimented with more or less research time, or in some other way have found a mix of preparation and publication that maximizes the value of their own efforts, in their own eyes of course. It seems entirely possible that asking them to do more writing would result in substantially lower quality teaching, otherwise they would have made that trade themselves. On the other hand, asking them to do substantially more class preparation would result in less writing without substantially advancing their teaching effectiveness. We should not be surprised to find that each faculty member reaches his or her optimum at a different mix of teaching and writing. This sets up what might be viewed as a restriction-of-range problem. It is possible that the data would not reveal any correlation for the group even though a complex relationship would indeed exist for each faculty member.

In short, it seems likely that writing and teaching are complements at low levels of writing and substitutes at high levels of writing, and determining whether any particular faculty member has increased his or her writing to the point that further increases will reduce teaching quality is a very tricky business.

Jeffrey E. Stake

In Defense of Teaching Evaluations as a Valid Measure of Quality

Drawing upon a large representative sample from 19 ABA-approved law schools, Ben Barton’s study found no statistically relationship between various measures of scholarly output and teaching effectiveness.  If this result is correct, then much of the legal academy’s discourse on “law school quality” is fundamentally flawed.

Rather than let go of old and comfortable ideas, many law professors will look for a basis to dispute the study’s findings. As noted in Ben’s last post, the critique of his study that will probably enjoy the most currency is the claim that law students are incapable of evaluating effective law school teaching.  As a result, teaching evaluations are an invalid measure of instructor quality.

In Ben’s defense, I believe he has ceded too much ground.  There are at least three reasons why the critics of teaching evaluations are wrong. 

Teaching Evaluations

Thanks so much to Jeffrey for his extremely thoughtful response to my study.  There is a lot there, so I'll just respond in chunks, and start by ceding his conclusion: "the finding of no correlation does not eliminate the possibility of a complex causal connection."  That is likely true of any correlation study, and is certainly true here.  Any correlation study is only as good as the data, and even with perfect data a correlation study can only report a raw finding of either positive, negative, or no correlation.  As I note in the paper it is dangerous to extrapolate too much from a correlation study, and I certainly agree that I can't rule out some hidden complex causal connection.

Nevertheless, I do feel obligated to mount a defense for the much maligned student evaluation.  I have three basic lines of defense.  The first is practical.  For better or worse, teacher evaluations are the only viable way to measure teaching effectiveness for a study of this breadth.  My other choices were exceedingly unpalatable: 1) attempt to gather peer evaluation data, which is rarely if ever expressed numerically, and would also almost certainly not be provided by the host institutions; or 2) use some type of personal subjective measure of teaching effectiveness, potentially requiring me to personally visit classes and make my own determination on teaching effectiveness.  In sum, these data are not only the best data, they are the only usable data.

The second is empirical.  While there are various studies showing biases in student teaching evaluations (although I am not aware of a study that shows the biases Jeffrey mentions), there are other studies showing that student evaluations are reliable and valid, and that student evaluations correlate with other measures of teaching effectiveness, including  peer evaluations and tests of student retention.  My favorite overview is Herbert Marsh's 1987 publication "Students’ evaluations of University Teaching: Research Findings, Methodological Issues, and Directions for Further Research."   Theall and Franklin have a more recent book called "The Student Ratings Debate" that provides a good overview of the controversy, and a chapter entitled “Looking for Bias in all the Wrong Places" that updates the state of the empirical studies and provides further support for student evaluation validity.

My last line of defense is, sadly, anecdotal.  It was not that long ago that I sat in a law school classroom and evaluated my law school professors.  At that point I had experienced approximately 17-19 years of schooling, and I think I was a pretty decent judge of good teaching.  I had certainly seen my share of both over the years.  I was also poised to take on the responsibility of becoming a lawyer and representing actual clients in court.  Nevertheless, many members of the legal academy assume that I was unqualified to determine whether the teaching I received was effective, and even guess that I filled out my forms based on a popularity contest, or the ease of the course.  This strikes me as somewhat paternalistic, and an underestimation of the abilities, sophistication, and discernment of our students.  I will also note, for the record, that I am a solidly middle of the pack teacher at Tennessee based on my evaluations and I accept that as a fair evaluation.

Teaching - Writing Correlation

I have a few comments on Benjamin Barton’s conclusion that there is no correlation between prolific writing and effective teaching.

First, Barton uses student ratings as the only measure of teaching effectiveness. He does not establish the degree to which student ratings are useful measures of teaching effectiveness, if effectiveness means that students learn facts and processes of thinking that are more important to learn than other facts and processes they might learn with equal investment. The connection has employed educators for decades and the correlation between student ratings and teaching quality is too weak, especially across various methods of teaching and types of classes, to justify using ratings as the sole basis for policy.

Second, student ratings may be biased. Suppose that students shift their evaluations upward for teachers who write more, as well as for teachers who teach better. A teacher could shift time away from teaching to writing, reducing the quality of teaching, without that reduction in quality being reflected in his ratings. Or, just to take the other side, suppose that people who write more are more aware of uncertainty and present ideas less definitively, lest the students get the wrong idea. As such a teacher writes more, her actual teaching quality rises, but her increasing indeterminacy prevents her ratings from rising. Or suppose that teachers who write more teach worse but teach more upper level elective courses which students tend to rate more highly. In all three examples, the true correlation is hidden.

Third, suppose that for teachers who publish little, writing more would improve teaching, while for teachers who publish much, writing less would improve teaching. Aggregating the groups would make it look like writing more had no effect on teaching, even though it had an effect within both subgroups. Lindgren’s focus on subpopulations might better reveal actual relationships within those subgroups, although such results should not be generalized to the whole population of teachers.

Fourth, a meaningful relationship between teaching and writing within a small subgroup might be masked by all of the data in the rest of the spectrum. For those who write a lot, shifting even more time to writing might decrease teaching quality, yet that effect might be masked by the large group of teachers who write little and for whom increasing writing does not detract from teaching. For some faculty members, increasing publishing does not mean decreasing teaching preparation. For others it does. If the former group far outnumbers the latter, the lack of a relationship within the former may mask the relationship that does exist in the latter group.

What almost *has* to be true? If a professor is working the maximum number of hours already, spending more time on teaching means less time on writing. How many teachers working to the limit have found that teaching is best when NO time is spent preparing for class? Starting from zero class preparation time, teaching will be improved more by spending five minutes on class than by spending that same time writing. Some class preparation is better than none.

However, by encouraging rigor and allowing analysis to be carried beyond what can be held in memory, writing often teaches a writer lessons he would not have learned had he not committed ideas to paper. It seems likely that many teachers will teach better when they spend at least a little time writing for publication. Even when only teaching quality matters, some writing beats none.

These points might form the foundation of a better model. There is a non-zero optimum level of research, and moving away from it will reduce teaching quality. To test it, we might look at those who write the most and try to see whether writing even more reduces student ratings. Looking at those who write least, we might try to determine whether writing more improves student ratings. Increased writing may have different effects on teaching at the two ends of the spectrum, and aggregating all teachers could wipe out evidence of those effects. In short, the finding of no correlation does not eliminate the possibility of a complex causal connection.

Jeffrey E. Stake

Thanks so much

First of all, thanks so much to ELS for taking the time to have a blog forum on my empirical study of the relationship between teaching and scholarship at American law schools, and for Bill and Jeff for agreeing to comment on the study.  I can say at the outset that there were many times over the last three years that I wondered whether I would ever make it to the finish line on this project.  The biggest problem was gathering the teaching evaluation data.  The University of Tennessee's data is publicly available on line (although in a very discrete and hard to find location), so I thought I would have little difficulty gathering data from 19 other American law schools.  As it turned out I needed to contact every law school dean and associate dean, and then file 12 State FOIA requests before I reached 19 participating schools.  I had originally aimed for 20 schools, but I would have had to file a law suit to make 20, so I called it a day at 19.

The difficulties I faced in gathering the data actually helped prepare me for my counter-intuitive finding of no correlation, and also for some of the critiques I've received.  As you might guess, anyone who thinks that teaching evaluations are bunk has little use for the study.  As you might also guess, these same folks have not been shy in sharing that opinion!  I've presented the paper at 3 of the participating schools and the SEALS conference, and have gotten some great feedback.  Nevertheless, I always face at least a partially skeptical audience.  Thus, I am most interested to hear what Jeff and Bill (and any Blog readers or commenters) have to say on the subject.  I am also quite interested in how people view the ramifications of the study.  If it is true that teaching and research are not correlated, what (if anything) does that tell us about law school hiring, management, and rankings? 

Can We Measure the Relationship Between Teaching and Scholarship in Law Schools

Today, the ELS Blog is pleased to host a one-day forum on the relationship (or lack of relationship) between teaching and scholarship in law schools.  The forum is prompted by a recent empirical study by Benjamin Barton of University of Tennessee Law, which has received a lot of play in the blogosphere (see, e.g., Comglomerate, Prawfsblawg, Legal Theory Blog, Leiter, and Stuart Buck).   The buzz, of course, is focused on Ben's primary finding that there is no correlation between effective teaching and scholarship.

This topic reflects a long-running debate within the legal academy, most of which occurred without data.  In 1998, Jim Lindgren and Allison Nagelberg began to fill this void with a small scale study based of faculty from three relatively elite law schools which documented a modest positive correlation (.20) between scholarly productivity (as measured by citation counts) and teaching (as measured by student evaluations).  See Are Scholars Better Teachers? 73 Chi. Kent L. Rev. 823 (1998).   Barton's study is much broader in scope.  It includes four years of teaching evaluations for 623 faculty members at 19 ABA-approved law schools.  In addition, it includes five discrete measures of scholarly output:  academic-oriented books and articles; practice-oriented books and articles; total publications; citations per year; and total citations.

As Ben notes in his conclusion, the lack of a empirical relationship between teaching and scholarship creates quite a quandary for legal education, especially as law school competition increasingly takes the form of attracting and retaining more productive scholars.   

To probe this issue further, the ELS blog is pleased to welcome Ben Barton, who will open the forum with some additional remarks on this research.  To provide commentary on Ben's study, we also welcome Jeffrey Evans Stake (Indiana Law), who is particularly well-suited for this forum.  In addition to his innovative work on law school rankings (see, e.g.,  81 Ind. L. J. 229 (2006); 80 St. John's L. Rev. 301 (2006)) and the Rankings Game website, Jeff is an award-winning teacher who has done extensive service work to improve the teaching evaluation process.