Empirical Legal Studies

Special Oxford University Spring School course, June 18-20

Chris Zorn has asked us to post the following announcement:

The Department of Politics and International Relations at Oxford University will be offering a special Spring School course, June 18-20, 2007.  The title of the course is "An Introduction to Event History Analysis," and the course will be immediately preceded by a "Crash Course in Stata."

The deadline for registration is May 18, 2007.  Details on the course can be found at here.  Note that the registration fees vary slightly as a function of whether or not a participant's institution is an ECPR member.  Many U.S. institutions are; check with members of your political science department if you're not sure.

Seems like a nice way to combine business with pleasure.

ELS Listserv

Thanks to former guest blogger Carolyn Shapiro (Chicago Kent), the members of the newly created ELS Listserv are introducing themselves to each other.  There is a strong untenured tilt to the members, several who joined to facilitate an empirical project that draws on their substantive research fields, some really diverse work experience and educational backgrounds, and a surprising trans-continental representation.   Overall, a very interesting group.

If you want to get in on the ground floor, email Carolyn.

CORRECTION:  The listserv is untenured, pretenured, and graduate student by design.  When some of the members get tenure, the issue may be revisited.

Laptop Debate: Effect on Exams versus Class Discussion

Peter Lattman at the WSJ Law Blog is asking readers whether Professor David Cole's no laptop policy is a good idea.  Cole, who teaches at Georgetown Law, published an anti-laptop essay in the Saturday Washington Post.  Like Professor Cole's students, the comments appear to be favoring the ban.  But there are some dissenters.

Perhaps this is the knotty empirical question that lies beneath this controversy:  What if final exam performance is maximized by laptop transcription, but the quality of class discussion is maximized by no laptops?   I think there may be some truth to this.  Many students claim that laptop notetaking helps them better distill the information after class.  If so, it is quite a conundrum. 

Any thoughts on a research design to test it?

The Influence of Oral Arguments (With Update)

I would like to draw your attention to a paper, see here, that uses the oral argument grades assigned by Justice Harry Blackmun to examine various issues relating to the effectiveness and influence of Supreme Court oral arguments. In the paper, which is forthcoming in Washington University Law Review, Tim Johnson, Jim Spriggs, and Paul Wahlbeck examine a random sample of 539 cases decided by the Court between 1970 and 1994 (spanning Justice Blackmun's tenure).

The aspect of the article that is most interesting, at least in my opinion, is the part of the paper that discusses those attributes of oral advocates that are most influential to the oral argument grades assigned by Justice Blackmun. Using such variables as whether the solicitor general is personally arguing the case, the amount of experience for the advocate arguing before the Court, and whether the advocate is an elite Washington attorney, former Supreme Court law clerk, or law professor, the authors run an ordinary least square (OLS) regression to determine which of the aforementioned factors most influence the oral argument grades assigned by Justice Blackmun. The results are not surprising as the personal appearance of the Solicitor General generally leads to a higher grade, and the presence of a former Supreme Court clerk or elite Washington attorney are also influential on the quality of an oral argument.

I do, however, question the grader in this case. Justice Blackmun appears to have been an especially harsh grader and the grades he assigned to several prominent legal figures are quite surprising. In the case of now-Chief Justice Roberts, who I believe is one of the great oral advocates of the past two decades, Justice Blackmun gave him only "average" grades in the twelve cases in the sample. In fact, in one of them, he said that Roberts did "better than usual," earning a grade of six on an eight point scale. When I clerked at the Court, I always made a point of being in the courtroom for Roberts' oral arguments as I always felt that there was much to be learned from his exceptional advocacy style and level of preparation. In addition, Blackmun gave only average grades to Justice Alito, and Justice Ginsburg did not fare particularly well in the sample either.

Note: Cross-posted at SCOTUSblog.com.  Please also feel free to comment on my post on the challenges of data coding below, and I hope to have a post completed early next week on the use of descriptive statistics in empirical scholarship.  I would like to thank Jason for permitting me to hang around for another week.

UPDATE: In response to my cross-post on SCOTUSblog.com yesterday, Howard Bashman wrote an "On Appeal" column for Law.com discussing the paper and explaining the importance of having experienced appellate attorneys arguing before the Court, see here.

Is there a Coastal Bias in the Lawyer/Judge US News Survey?

Over at the Legal Professions Blog, my good friend Jeff Lipshaw, equipped with an excel spreadsheet, a map, and a ruler, has taken on the perennial claim that the Lawyer/Judge input variable in U.S. News contains a "coastal bias."   His analysis says no, but Jeff has asked for some help. 

Using a simple OLS model, my dependent variable is 2006 Lawyer/Judge Reputation.  My independent variables include Academic Reputation, 75th percentile LSAT (which is good proxy for total rank), a west coast dummy (Arizona, California, Oregon, and Washington; 27 schools = 1) and an east coast dummy (Delaware, Maine, Maryland, Massachusetts, New York, New Hampshire, New Jersey, Pennsylvania, Rhode Island, Vermont, and Virginia; 54 schools = 1) .  Here is a summary of the results, which corroborate Jeff's theory:

Obviously, the signs for both coasts are negative (i.e., no systematic boost for either coast), and the p-value for East Coast implies a statistically significant penalty for being an east coast school.  Perhaps schools far away from the major east coast legal markets are destine to seem better than the nearby law schools that firms know well but tend to run down.   The adjusted R-squared for the model is 75.7%.

We might be able to specify a better regression model, or recode with a better definition of "coastal", or correct for some minor heteroscedasticity.  But anyone who thinks they can salvage the bias theory is probably kidding themselves. 

Jeff, you should plan on spending the summer in Ann Arbor at the ICPSR statistics boot camp.  You clearly show promise. 

Dummy Variables in Cross-Country Regressions

I post the following at the request of a current student at Marquette.  If you have any insights for him, please post a comment or email me at jason.czarnezki@marquette.edu.

"I am writing a paper on economic growth and want to look at the effects, if any, that the mixture of civil law and common law in Louisiana has had on that state's rate of growth.  Theory and past research using cross-country regressions show that nations with civil law origins have lower rates of economic growth, all else constant.  I want to use a cross-state comparison (LA, MS, AL, and AR) and include a variable that captures the affects of LA's mixed legal system of the state's rate of growth.  Most of the research I have looked at uses a simple binary choice dummy variable to distinguish the legal origin.  However, as you probably know, this dummy can pick up influence from any number of variables not included in the model and that are unrelated to legal origin.  Therefore, I have been trying to come up with some way to capture the legal origin distinction, but with a proxy variable that provides a greater quantitative distinction for legal origin than the dummy.  Some quantifiable measure that would theoretically move with the ebb and flow of civil law influence on the state's legal system, which has waxed and waned since LA became a state.  There may be no real good solution for this problem (which I imagine is why dummy variables exist in the first place), but I would appreciate greatly any helpful comments or ideas." 

The Challenges of Data Coding

Many of my recent projects have required quite a bit of data coding.  My review essay in this month's issue of the Texas Law Review, for instance, required the review of approximately 20,000 pool memos from the Blackmun papers in order to evaluate the characteristics and possible influence of the Supreme Court's cert pool.  Another project, which I hope to co-author with Tim Johnson (University of Minnesota Political Science Department), will examine the factors that influence the certiorari votes of each of the Justices on the Court during the latter part of Justice Blackmun's tenure on the Supreme Court (1986-1993).  Finally, I have been collecting data for almost a year for an article examining the reasons behind the Supreme Court's declining plenary docket.  Among other things, the latter two papers have required coding the cert votes of every Supreme Court Justice in every case (with at least one vote to grant or join-3) from 1986-1993.   The data set includes between 2,000-3,000 cases, and approximately 20,000 total votes.

The amount of data collection required for these various projects is extensive, and I can now say that the collection and coding of the data has been one of the most difficult aspects of empirical research.  The first challenge, of course, is figuring out exactly how you want to code the data, and which variables should be included in the data set.  Without careful consideration, you may quickly find yourself well into the project, and having to review and reenter much of the data because an important variable has been unintentionally omitted.  I have found that, when examining and studying courts, the importance of some variables is often not self-evident, and thus it pays to be overinclusive in the amount of information and number of variables coded on the front end.   

The second challenge is competence.  Law students, unlike graduate students in the social sciences, are not trained in data coding. Very few have quantitative backgrounds, and even fewer have experience working with large data sets and statistical packages such as SPSS and Stata.  As a result, my empirical research has required me to train and review the work of my research assistants quite closely, which takes time away from the research itself.  Thankfully, I now have a research assistant that has become quite proficient at coding and interpreting data, but I am coming to grips with the fact that he graduates at the end of this year and I have yet to find a suitable replacement.  Again, unlike graduate students, law students are barred from working for professors during their first year, and thus it is only possible to work with them for a maximum of two years, hardly enough time for them to gain competence and speed in data coding.  One possibility, of course, is to hire graduate students in the social sciences, but for some reason beyond my understanding, it is far more costly to hire research assistants from outside the law school (as much as three times as expensive).   I still have yet to find a suitable answer to the second issue, and I wonder how my colleagues in the legal academy deal with this question.  Comments to this post are therefore most welcome.     

Interdisciplinary Scholarship and Co-Authorship

I noted yesterday that law schools have really focused on interdisciplinary scholarship in recent years, especially when hiring new faculty.  But it is also fascinating that many law schools have failed to adjust to the type of scholarship produced by such scholars.  Many newly-hired faculty have Ph.D.s in such areas as economics, history, or political science, and are accustomed to the promotion and publishing standards of the disciplines in which they did the majority of their post-graduate work, not those of the legal academy. 

For example, my experience in political science is that it is rare to see articles that are produced by only a single author.  Some papers have five co-authors or more, and most have at least two.  Yet my conversations with interdisciplinary scholars at many law schools seem to indicate that co-authored works are discounted, especially when more than two authors are listed on the piece.  Another prominent example are books, which often take far longer to produce than an article, yet again many law schools do not give such works the credit that they deserve. Yet as we move into a new era of interdisciplinary research, shouldn't our standards adjust to compensate for the new focus? 

As a young scholar, I have passed on or postponed a number of interdisciplinary collaborations due to my own uncertainty about how the works produced would be viewed both within and outside of my institution.  Yet I also know that collaborations with my colleagues in political science or history would result in works that are better and more insightful than what I can accomplish on my own.  My own experience with co-authorship is that each of the authors puts in about 75% of the work of a solo-authored project, primarily because collaboration results in many discussions about the work with your co-author, and nearly all of the editing and thinking that would be required of a solo piece.  If properly split up, the only aspect of co-authorship that is easier is that you do not have to all of the research and writing of the piece; instead, that obligation is shared by both authors.  I greatly enjoy and have done both types of projects, solo and co-authored, but the institutional incentives seem to clearly favor solo pieces even though the legal academy has made this steady march toward interdisciplinary work.

Tomorrow, I hope to have a post commenting on the challenges of coding empirical data.          

Law and Politics

My post this morning was designed to consider the challenges posed by increasing interdisciplinary scholarship in law schools aside from the traditional critique that legal scholarship is no longer useful for judges and practitioners.  The rise in the emphasis by law schools on interdisciplinarity has thankfully been accompanied by an increase in resources available for the interdisciplinary scholar. 

One valuable resource is this blog and others like it.  ELS, for example, brings together scholars from a variety of disciplines, including political science and the law.  As someone who has this site bookmarked and reads it relatively often, I find papers posted here that would otherwise not cross my radar screen. 

Another is SSRN, which as you know serves as a repository for papers from a variety of disciplines.  In an effort to promote greater awareness of scholarship in the areas of political science and the law, Tim Johnson (University of Minnesota Political Science Department) and I have started an SSRN e-journal on "Politics and the Law," which is now on the Legal Scholarship Network and will be cross-listed on the Political Science Network once it is created.  If you have interest in the intersection between politics and the law, I encourage you to subscribe to this newly-created journal.  Similarly, other disciplines, such as legal history, have created e-journals for much the same purpose.  The Experimental and Empirical Studies e-journal is likewise excellent for empirical scholarship.

Finally, as I mentioned this morning, conferences promoting interdisciplinary scholarship are becoming more widespread.  A great example is the Empirical Legal Studies Conference, which features papers by scholars from a variety of disciplines.  Although I had a conflict with the conference last year, it is my hope to at least attend the conference this fall and possibly even submit a paper for consideration.  The line-up of presenters last year was most impressive, and I expect that it will be equally, if not more, remarkable this fall.

I am sure that a number of other resources exist that I am unintentionally omitting, and would welcome other suggestions in the comments to this post.  In a post tomorrow, I hope to continue the discussion by considering the relationship between interdisciplinary scholarship and co-authorship, and whether interdisciplinary scholars may be at a disadvantage due to law schools' traditional discounting of co-authored works. 

Interdisciplinary Legal Scholarship

In recent years, interdisciplinary legal scholarship has become more common, and many law schools seem to be placing greater emphasis on recruiting scholars with an interdisciplinary bent.  Although I am by no means familiar with many of the "law and [insert field]" areas of scholarship, my own research on the Supreme Court requires familiarity and facility with political science scholarship on judicial decision-making.  Although the environment seems to be slowly improving with a greater number of political scientists publishing in legal journals, there still seems to be a disconnect between the work of legal scholars and political scientists, with many in both fields ignoring the scholarship of the other.  I have often wondered about the reasons for this phenomenon.

One simple possibility is that political scientists and legal scholars have historically published in journals that are housed in different electronic databases (or in the pre-computer days, different libraries).  For instance, most political science journals are not available on WESTLAW or LEXIS, but can be found on such electronic databases as JSTOR and Academic Search Premier.  Of course, in law school and private practice, lawyers are taught to rely heavily on WESTLAW and LEXIS, but have little familiarity with the other electronic databases.  Thus, a "quick search" on WESTLAW will not reveal "on point" scholarship from other disciplines.

Another possibility is the insularity of each of the disciplines.  For many years, the work of legal scholars consisted primarily of doctrinal work and normative recommendations stemming from theoretical (rather than empirical) policy arguments.  Political scientists have often viewed the work of legal scholars as insufficiently methodologically rigorous, even when we have ventured into empirical research.  To some degree, that might even be true today, although the amount and degree of collaboration between political scientists and legal scholars has vastly improved.  Meanwhile, legal scholars have criticized political scientists for failing to see the nuances in the law, and focusing too much attention on institutional and external factors to explain judicial decisions.  Legal scholars also criticize political scientists for rigid adherence to models of judical decisionmaking without explaining the outliers in the data or the imperfections of the models.    

I am sure that there are other possibilities, and I don't mean this post as a criticism of either discipline, but I do believe that greater collaboration between political scientists and legal scholars will result in more complete explanations of judicial behavior.  A legal scholar that focuses solely on doctrine may be missing half the story, while a political scientist that ignores the law might be missing the other half.  Having studied the Supreme Court and other federal courts from several different perspectives (including from the inside), it seems clear to me that judicial decisionmaking is a product of a host of factors, some of which political scientists are better able to evaluate (such as external political constraints) and others that are more within the expertise of legal scholars (such as stare decisis and nuances in doctrine).

I see it as a positive sign that political scientists now participate widely in legal symposia, and legal scholars now present papers at political science conferences.  I also know that many of my colleagues now run a search (as I do) in Academic Search Premier or JSTOR before writing an article on the court system or judicial decisionmaking.  In addition, I can say without hesitation that I have personally learned a great deal from discussions with my political science colleagues and from attendance at presentations sponsored by Minnesota's political science department.  The next step, I think, is to see greater co-authorship between legal scholars and political scientists.  Even so, I see a healthy trend developing.