Empirical Legal Studies

Where's the Theory?

Yesterday, I asked readers to consider the relative merits of aggregate v. individual level analysis. Regardless of the empirical approach, our analyses must be framed by theories about individual behavior. To the extent that we are concerned about developing an understanding of the choices of either individuals or organizations, we need to make certain explicit assumptions about what motivates choices.

Interestingly enough, it is here where I find most of our empirical work to be especially lacking. Despite our ability to generate a great many interesting empirical insights, we seem not to be as concerned as scholars in other disciplines with laying the building blocks that support our expectations.

I do think that we know a great deal about, say, the behavior of judges, especially appellate judges in the United States. Nevertheless, I’m not sure that we have anything like a “theory of judging.” Several years ago, Larry Baum wrote a wonderful book, The Puzzle of Judicial Behavior, in which he described what the various motives of judges might be and how their personal goals and professional goals might interact with what judges can do in different institutional settings. One of his conclusions is that we need to be more cautious about imputing motives to judges.

What are the consequences of our collective inattention to theory? If we simply want to predict behavior, then the ill-effects are not really that severe. If we want to understand and explain that behavior, then I think we should be more guarded. We may misdiagnose much of what we observe.

Perhaps one reason for our lack of theory is that, since many of us actually are what we study -- i.e., we are often lawyers studying the behavior of lawyers -- I suspect that the kind of formal training and informal socialization that takes place within the legal community leads to bit of overconfidence; that is, since “we know how lawyers really think and behave,” our empirical work can proceed according to long-internalized assumptions. Those assumptions may be quite correct, but they may also fail to consider other plausible options.

Imagine, for example, studies of congressional behavior that were authored by members of Congress. On the one hand, I would take very seriously what those analyses had to say, since they would be informed by the perspective of individuals who possessed a detailed knowledge of both the mechanics of legislative institutions and the various incentives and constraints that they face in making difficult decisions. On the other hand, I would almost certainly be circumspect about the results, since they would be based upon assumptions that were heavily informed by the personal experiences and biases that necessarily come with it.

Now, I am not a lawyer, and that surely limits what I am able to do. In fact, I’ve been humbled on several occasions by lawyer friends who have pointed out the limitations of some of my work by drawing upon a perspective that I simply don’t have. And this has saved me from embarrassing errors I might otherwise have made. At the same time, those of us who are not lawyers may have a more ready knack for generating a wider range of assumptions -- some plausible, some less so -- that might account for what we observe.

Despite this flexibility, none of us has had any greater success in specifying a general theory of why judges do what they do. To be sure, we often produce research with provocative and surprising results, but it is not at all clear that we share a basic core of ideas about the choices judges make. As a result, we probably have a harder time reconciling disparate findings or sorting out the relative importance of the causes that we think are at work.

I’m certainly not prepared to propose any general theories about judicial behavior, but I do think that we should exercise greater care and be more explicit about the assumptions that go into our empirical work. It will surely improve the discourse between various communities who study the law and enhance our understanding.

Lumpers v. Splitters

One of the interesting questions of empirical analysis is whether to approach a research question from a macro-level or a micro-level perspective. Should we aggregate data, ignore relatively small differences, and look for the big picture? Or should we look at the individual level, take account of particulars, and try to leverage the differences between cases in order to increase the sophistication of our explanations?

The approach one takes, I suspect, has a lot do with the type of question one hopes to address, but I’m particularly interested in the tradeoffs that exist between the two for those of us who are interested in formulating an empirical view of law and courts. Is one approach inherently preferable for what we study? Does one approach make quantification more problematic?

My impression is that those of us who do empirical work on the law have, on balance, been lumpers more often than splitters. Certainly my fellow political scientists do lots of convenient aggregating of data -- across judges, courts, states, and years, for example -- to test various notions about the intersection between law and politics. As a consequence, I think we have some very good evidence relating to, among other things, judicial decision making and the relationship between courts and other political actors.

In treating courts as political institutions, however, we are sometimes criticized for failing to consider more specific legal factors, such as doctrinal tests, legal customs, methods of statutory interpretation, and other variables that are regarded as significant for explaining what courts do. The consistency of these criticisms should, I think, give us some pause.

Given the growth of empirical legal scholarship among people with fairly diverse training and orientations, I am interested to see how both lumping and splitting will be brought to bear to offer some new insights. For instance, I can imagine that splitters, who see the need to test the kinds of individual differences that lumpers often ignore, will end up devising refined explanatory models with which the lumpers will have to come to terms. Indeed, an interesting body of work is already developing along those lines.

Obviously, I’ve oversimplified things a good deal, but my point, as I start off this week, is to ask others to think about which analytic approach will yield the biggest benefits and when. Which questions of law and courts are better handled from the macro-perspective, and which are more profitably addressed at the micro-level?

My thanks to Jason Czarnezki for inviting me to serve as a guest-blogger.  I look forward to participating throughout the week.

Guest Blogger: Kevin McGuire

This week's guest blogger is Kevin McGuire, an Associate Professor in Political Science at North Carolina at Chapel Hill.  Kevin teaches courses on the U.S. Supreme  Court and American constitutional law.  His current research interests include the influence of public opinion on the justices’ decisions, the content analysis of the Court’s written opinions, measuring the ideology of the Court’s case outcomes over time, and compliance with judicial policy.

Empirical Work on Transaction Costs

The Coase theorem holds that in the absence of transaction costs, all allocations of property are equally efficient because affected parties will privately bargain to correct any externality.  The familiar rejoinder by people who don't like law and economics is that transaction costs are never zero--but that was exactly Coase's point.  Since transaction costs distort or shape virtually all outcomes, effective social policy depends upon our understanding of real world contexts.  In short, the natural upshot of the Coase theorem is a comprehensive empirical research agenda on a wide range of legal and nonlegal disputes--enough work to keep several generations of researchers busy.  (Granted, those of us without a Nobel Prize under our belt have been somewhat slow to catch on.)

With this understanding of Coase in mind, this paper by Barak Richman (Duke Law) and Jeffrey Macher (Georgetown Business) caught my eye:  Transaction Costs Economics: An Assessment of Empirical Research in the Social Sciences.  Here is the SSRN abstract:

This paper provides a comprehensive review of the empirical literature in transaction cost economics (TCE) across multiple social science disciplines and business fields. We show how TCE has branched out from its economic roots to examine empirical phenomena in several other areas. We find TCE is increasingly being applied not only to business-related fields such as accounting, finance, marketing, and organizational theory, but also to areas outside of business including political science, law, public policy, and agriculture and health. With few exceptions, however, the use of TCE reasoning to inform empirical research in these areas is piecemeal. We find that there is considerable support of many of the central tenets of TCE, but we also observe a number of lingering theoretical and empirical issues that need to be addressed. We conclude by discussing the implications of these issues and outlining directions for future theoretical and empirical work.

Information Technology for Data Collection

Many of us doing empirical legal research are involved in data collection projects of various types. Until very recently, my research assistants and I wasted a tremendous amount of time duplicating work and coordinating amongst ourselves. Recently I have learned -- from some of our excellent IT staff and some savvy graduate students -- about useful systems we all can use to manage our data collection projects.

The traditional approach is to hire teams of law or graduate students to collect and and code data, perhaps reading cases and coding a number of variables. What I had done in the past is to have the students enter data in either Excel or Stata, and then return their portion of their data to me when complete. Then, a research assistant would create a master dataset from all of the various sources. This approach is highly inefficient, and provides many opportunities for human error.

A better way to perform this sort of data collection is to create a web-based data collection portal that research assistants can use to input coded data. One advantage of this is the ability for multiple research assistants located anywhere to read from and write to the same data set. Systems can also be built to allow for various levels of integrity checks, supervision, etc. And, when complete, it is straightforward to build systems to deliver data in whatever form you need.

Here at Wash U we have built a system for a project my colleague Margo Schlanger has undertaken called the Civil Rights Litigation Clearinghouse. You'll be hearing about the substance of this exciting project on this blog and elsewhere starting in about a month. A crucial facet of this project was creating an information system to catalog and subsequently disseminate a diverse collection of case documents. And, germane to this post, to allow research assistants and the data owner, Margo, to code various aspects of the cases, documents, and court officials.

Below are some screenshots of the administrative back-end of this site. Using this environment, research assistants enter data about a case using actual court documents collected by the data owner. In this case, entered data can include the case name, the document title, the document type, citation etc.

click to enlarge

The data owner plays a governing role in overseeing the work done by their research assistants. It is important that a workflow and auditing system is present so information entered into the collection may be reviewed and modified where necessary. In the below screen the data owner views the status of an existing case. Here the case summary is in progress; the approver may edit what the student wrote and when ready approve the changes. Such measures allow data to be phased through an identified process and allow the public display of items to be granularly controlled.

click to enlarge

This custom information system has made the full maturation of this very ambitious data collection project feasible. Without a consistent and stable method of collection and interpretation, large data sets often get obfuscated and disjointed, thus rendering them difficult if not impossible to share in any meaningful manner.

Obviously planning for and organizing such an intricate piece of technology requires the right sort of collaboration. It is vital that appropriate subject matter experts are consulted as to avoid missteps in the planning, which is so often the most crucial step to any technology undertaking.

In this case, the base-technologies were quite standard; apache webserver, php and a mysql back-end. In addition to the hard technologies, it is helpful to have people skilled in information architecture, site design, web usability, database management, and obviously web development. And this is of course all in addition to the data owner and their collection, without which, the endeavor would not be necessary.

Another common task the people frequently undertake is harvesting data from various websites. This can be done by humans using point-click-copy-paste. But that, too, is inefficient and opens up lots of opportunities for human error. It also makes it difficult to replicate studies conducted by others, and to update existing datasets. There are a variety of scripting languages, including perl (using LWP::UserAgent for fetching and HTTP::Request::Common for posting forms) and python, that can be used to harvest data from websites and turn it into useable datasets. The Snoopy php module also works well for these purposes.

Using these technologies is more expensive in terms of time and resources at the start-up phase of a research project. But the efficiencies that can be captured are quite significant.

What Makes a Paper "Interesting"?

Over the years I've sat on a number of search committees in political science departments and other departments in the college. This year I'm serving, for the first time, on the entry level appointments committee in the Law School. One conversation that oftentimes comes up in our discussions is whether a paper is "interesting" or not.

What do people in legal academia view as "interesting" work. I, personally, have standards I use to judge whether a paper is of theoretical or substantive importance as relates to the social scientific literature. And, editors always ask me to assess the research design, data, and methods of any piece I'm refereeing. But I'm having a hard time figuring out what makes a piece, whether empirical, theoretical, doctrinal, or something else "interesting" to the law review audience. Thoughts?

Warren Buffett as a Data Point

Shortly after Warren Buffett announced that he was giving away the vast bulk of his estate to the Bill & Melinda Gates Foundation, I heard a story on NPR that interviewed Buffett's granddaughter, Nicole Buffett.   Nicole recounted five specific pieces of advice that she received from her grandparents; and over the last several months I have wondered whether they are indeed the formula for a successful life (however one might define success). 

I ultimately concluded that the only way to empirically ascertain the benefits of these sayings was to follow them with earnest for a long period of time (many years, if not decades).   I passed along these sayings to my students on the first day of Corporations, so why not to ELS readers?  Here they are:

1. "Show up"
2. "Tell the truth"
3. "Pay attention"
4. "Do your best"
5. "Let go of outcomes"

I have found that if I do 1-4, #5 is much less difficult.    I may be able to provide more comprehensive data in a few years.

Measuring Case Complexity

Thanks a lot to Jason and the other editors for inviting me to blog this week. I plan to spend my time here tapping the collective wisdom of the blogosphere about what I view as a number of important questions. I'll also spend some time blogging about various tools we should all be using in our empirical work.

I'll start with this. One of things political scientists oftentimes include in statistical models of judicial decision making is a measure of "case complexity." In the Supreme Court context, the standard measure, as coded by Harold Spaeth, is to label a case "complex" if there was more than one legal issue or provision at play in the case. This may be not be an accurate measure of complexity. Many cases involve, say, a first amendment free speech issue as well as a due process or establishment of religion issue. Is this more complex than a tax case? Securities? Telecommunications? Surely not.

Any thoughts about how to systematically measure case complexity? Perhaps looking at the number of issues cited or the level of factual complexity might help. How could we do better?

A Return of the Wigmore Trial and Error Approach?

On Friday, law.com posted a story on a recently filed lawsuit against St. Thomas School of Law in Miami.   The plaintiff, a former student who is seeking to certify the suit as a class action, alleges that several St. Thomas administators violated federal anti-racketeering laws by admitting a disproportionately large 1L class with the intention of flunking out approximately 25 percent the following year (students with less than a 2.5 GPA).   

The school purportedly benefitted from this scheme by (a) collecting first year tuition from more 1L students, and (b) improving its anticipated bar passage rate for feature classes because students with low 1L grades are much more likely to struggle on the bar.   A St. Thomas official quoted in the story disputes the high attrition rate (only 40 of 330 1L's were dismissed) and called the revenue theory "illogical" because early dismissals reduce the school's income.

Setting aside the speculative nature of the lawsuit (and note that the ABA Section of Legal Education and Admission to the Bar has been named as a defendant on a negligent supervision theory), such a scheme would deliver clear economic benefits to many lower ranked schools.

Specifically, St. Thomas is a non-prestigious law school operating in Florida, which has lots of other law schools.   According data published in the ABA-LSAC Official Guide, the bottom quartile of entering class at St. Thomas is--relatively speaking--academically weak:   147 LSAT (12th lowest out of 190 schools) and 2.58 UGPA (lowest of out 190 schools).  Although these number bode poorly for bar passage statistics, this outcome can be mitigated by flunking out the bottom of the class after the 1L year.  After all, law school performance is a better predictor of bar passage than an LSAT/UGPA admission index.    Higher bar passage statistics could increase U.S. News rankings and applicant volume, thus setting in motion a virtuous upward cycle.  Moreover, the plan becomes revenue-neutral if the 1L entering class is significantly increased--a fact alleged by the St. Thomas lawsuit.

This system represents a return to the past, albeit the economic drivers are quite different.  As the modern law school hierarchy took shape in the second half of the twentieth century, one of the achievements celebrated by law school administrators was the very low 1L attrition.  The reason was simple; the LSAT had effectively identified those students who lacked the aptitude to become lawyers.  Prior to the LSAT, law schools relied upon the Wigmore method of trial and error, which winnowed classes based on first year exams.  See John H. Wigmore, Juristic Psychopoyemetrology--or How to Find Out Whether a Boy Has the Makings of a Lawyer, 24 Ill. L. Rev. 454, 463 (1929). 

Unfortunately, the Wigmore system seems destine to rise again as the number of ABA-approved law school continues to climb (+12 since the first complete U.S. News rankings in 1992), the applicant pool remains relatively flat or on the decline, and states continue to increase bar cut scores (at least a dozen states have done so in the last 15 years).  Indeed, I have heard of second-tier law schools meddling with 1L cut-offs in order to improve bar passage results.   Two observations:

1. Where is the disclosure?  One of the dangers here is that law schools don't bear the cost;  rather, it is borne by former students who incurred the time and expense of attending one year of law school before getting the boot.  At a minimum, academic attrition should be part of the data collected and disclosed by the ABA on an annual basis.  If the market is going to sort out this significant risk, it needs adequate information.
2. Can curricular innovation improve bar passage? An outsider to legal education might ask why curricular innovation is not the first line of defense for low bar passage.  Actually, there is data that this approach might actually play large dividends for struggling law students.  See Jellum & Reeves, Cool Data on a Hot Issue: Empirical Evidence that a Law School Bar Support Program Enhances Bar Performance, 5 Nev. L. J. 646 (2005) (exerpt reprinted in The Bar Examiner).

Perhaps the St. Thomas lawsuit will serve a catalyst for these two issues.

Guest Blogger: Andrew Martin

This week's guest blogger is Andrew Martin, Professor of Law and Professor of Political Science at Washington University in St. Louis.  Andrew is the founding Director of the Center for Empirical Research in the Law, and also directs the interdisciplinary Applied Statistics and Computation program.  Andrew specializes in political methodology, Bayesian statistics, and American political institutions. Much of his substantive research focuses on law and courts, particularly the Supreme Court and the Courts of Appeals.