ELS Past, Present and Future

In the 1930s, Yale Law Professor Underhill Moore, enlisting the young Lloyd Cutler as a research assistant (along with all of Cutler’s classmates – they had no choice), did a massive empirical study of how New Haven’s citizens responded to the presence of parking meters and sanctions for not feeding them. The resulting article, whose length was exceeded only by its tedium, may have doomed empirical legal studies for the next 20 years. Yet in many ways the project was well conceived. Parking behavior is a nice way of studying how ordinary citizens respond to mild sanctions. Parking behavior is easily observable, as are the costs of parking and the cost and likelihood of sanctions. Various independent variables, such as the value of a car (as an index of wealth) and the gender, race and approximate age of the driver may also be captured with considerable reliability, and enough cases may be generated to allow detailed analysis. But Moore lacked the technology to adequately analyze the data and to present his findings in ways that could be easily understood. This was more or less true of quantitative research on law, and most other areas, until the 1960s.

Outside of economics and psychology, the classic empirical social science offerings before that decade were mainly qualitative studies, books like William Foote Whyte’s Street Corner Society and Philip Selznick’s TVA and the Grass Roots, though a decade earlier, with the leadership of people like Paul Lazardsfeld at Columbia and the survey researcher Angus Campbell at Michigan and their students, modern quantitative social science was being born. By 1970, in all the social sciences except anthropology, quantitative research and formal modeling had replaced qualitative studies and informal theorizing as the hottest, and in some departments the only respectable, way to do social science. Yet except for a few celebrated works, like Kalven and Zeisel’s The American Jury and Wayne LaFave’s massive study of arrest processes, works that used numbers crudely if at all, legal scholarship was barely affected.  Underhill Moore’s ghost had not risen.

I think there were many reasons for this, including a law school culture that reserved its highest accolades for those most adept at analyzing legal doctrine, particularly the pronouncements of the Supreme Court; the view that those who had excelled academically at top law schools and clerked for the right judges needed no further training beyond, perhaps, a bit of practice experience before joining the nation’s leading law faculties, and, in many of the nation’s law schools, a two-and-a-half year, one grand article, path to tenure that did not provide the time needed for empirical data collection and analysis and did not recognize that good quantitative social science more often consists of the production of a series of fine bricks than the construction of one imposing wall.

I want, however, to focus on a different reason for the law school world’s relative inattention to social science.. Outside of certain special areas like antitrust, social science did not appear to have, and in large measure did not have, much to offer law. Even worse, law professors often saw in social science research caricatures of the enterprise they were engaged in. Nowhere was this more true than in the area most likely to attract the attention of law faculty – the study of Supreme Court decision making. Social scientists struggled long and hard to come up with models to predict how the Justices might vote in particular cases when most law professors, even those outside constitutional law, thought they could do as well or better predicting off the top of their heads. Moreover, the topic was neither interesting nor important from the law professor’s perspective. What mattered was not predicting judges’ votes, but understanding them and understanding and critiquing the doctrines promulgated in opinions.

In other areas, where social science might have informed the law, reliable data were not available, or methods were not up to the task of adequately analyzing the data, often because relevant data were categorical and methods for analyzing categorical data, beyond simple chi squares, were in their infancy.  When more complex methods of data analysis, as in time series regressions, were available and used, law professors often could not follow the methods and so distrusted the results, particularly if their implications conflicted with a professor’s intuitions about the state of the world or his policy preferences. Further distrust was sewn because social scientists who worked on legal issues often got the law wrong and thought their work was legally relevant when it was not. Moreover, social scientists were often perceived, sometimes justly, as trying to push an agenda. Bright lawyers, no matter how little they knew of formal methodology, often had a good intuitive sense of holes that existed in the logic of social science exploration, though this was too often coupled with a poor sense of what holes really mattered and what were less important.

Looking at things from the other side, social scientists interested in legal issues often had little interest in the specific concerns of law professors, and could be most turned off by those professors who seemed to their colleagues and regarded themselves as most friendly to social science. Often these professors did not want social science knowledge; they wanted social science facts. The feeling among social scientists – I remember it myself – was,  to exaggerate only slightly for effect, that the great law professor valued social science learning as a research resource . he saw the social scientist’s task as revealing the way the world worked and leaving it to him (professors were almost all men then) to decide what to make of it for law reform or other purposes.  But any social scientist worth his salt had his own research agenda, his own way of understanding institutions and his own perspectives, usually focused more on theoretical than normative concerns. Moreover, the sociologists of law knew that two of the pioneers in their field, Richard Schwartz and Jerome Skolnik, had spent time at Yale Law School and not been kept on. Thus there was, for the most part, considerable distance between the professions, and until social science became more useful to law faculty and law faculty became more respectful of social science  agendas their were few incentives to bridge it.

(My law school, Michigan, I should note, was in 1968 an exception. My Ph.D. work was regarded as a plus, equivalent to a Supreme Court clerkship, and I was hired before I graduated without any practice experience; Al Conard had been lavishly supported with law school funds to do a major empirical study of automobile injury compensation, and not only have I always felt respected, but I received tenure on the basis of a completed dissertation and three definitely not grand social science articles. However, Conard, was not a social scientist and had no interest in social science theory, and I cannot say how I would have been received had I not had a law degree and been able to analyze cases with the best of them.

The only school I can compare Michigan to is Yale which I visited in 1971-72 when they were advertising themselves as extremely interested in social science and had added the sociologist Stanton Wheeler to their faculty. Except for Wheeler and a few others, the description I give in the preceding paragraph describes my feelings of how Yale’s luminaries then regarded social science.)

The early exception to the estrangement of law from social science was economics, which because of its relationship to subjects like antitrust and utility regulation had long had a presence in law schools, and which began to be welcomed more broadly about the time I entered law school teaching. But the warm reception accorded economics was not because economists did empirical work, but because for the most part they didn’t.  Most economists in those early days, including many of the more celebrated ones, did not present law professors with messy empirical data whose implications for policy were uncertain; rather they offered neat formal models and clear specifications of how people were supposed to behave which provided an impressive tool for critiquing legal decisions and gave clear, though sometimes non-obvious, answers to questions regarding legal allocations and other contested legal policies. Some law professors found the (non-empirical) economics approach intellectually dazzling and/or ideologically satisfying, building a strong wing of our profession and training others in their perspective. Others ignored what was happening, only to find that the discourse of legal analysis changed sufficiently over time that giving students an economic perspective as at least one way of thinking about a problem or offering or contending with an economic perspective in policy scholarship is today almost as natural and unnoticed as speaking prose.

While I have described an estrangement, or more precisely a mutual lack of interest, between legal and social science scholars, it was not complete. If social scientists did not want to set their agendas by what law professors were most interested in, nonetheless they did work that was of interest to law professors and which the law professorate could use. Kalven and Zeisel’s American Jury a collaboration between a social scientist and one of the nation’s leading law professors was a magisterial work, which despite the methodological criticisms that could be legitimately made of it, changed the way legal academics regarded juries, and over time effectively undercut Jerome Frank’s “realist” denigration of the capacity of the common man to decide legal issues. Other work on juvenile courts, police arrest practices, plea bargaining and the like was eagerly read by people like my colleagues Yale Kamisar and Jerold Israel and fueled their commentary on and critiques of the nation’s criminal justice system. Contracts teachers came similarly to realize they could not teach their subject without referencing the work of Stewart MacCauley. Even the Supreme Court was citing social science, so law professors had to attend.

A lot of the empirical work that influenced law professors at the time was not quantitative, but as the social sciences became more quantitative and produced higher quality quantitative empirical scholarship, the work referenced became more quantitative as well. Moreover, almost from its start, in 1966, law professors were one of the major and most numerous constituent groups of the Law & Society Association which, in those early days was only about empirical research. Few of the quantitative articles in those early days were written by law professors, but they were being read by some of them. At the same time dual degree training was becoming more common and more law schools began to think that they had room for social scientists who were not economists.

Fast Forward. Today the situation is changed considerably. Dual degree holders are common, and the joke is that on some faculties unless you have a Ph.D. to go along with your law degree you won’t be hired. Ph.D. holders without law degrees are also common and increasingly trusted to teach core law courses.  Doctrinal policy scholarship regularly references quantitative empirical studies, and more and more law professors, trained and not trained in the social sciences, are doing empirical work. What has caused the change?

There are many reasons, but I think the core reason is that we, and by “we” I mean both social scientists and law professors, are doing better social science. This has been enabled by technological and intellectual developments, including the invention of new statistical techniques that take advantage of high-speed computing, the collection of and easy access to an increasing number of large, high quality data sets, increased experience with field experiments, and the availability of research support from NSF, NIJ and various private foundations. It is also due to the fact that the social sciences are sciences – each generation of discoveries builds on prior ones. As these sciences advance they test and build new theories and discard or revise prior ones. If the social sciences were not getting stronger and if social science learning was not more illuminating and reliable today than it was 30 years ago, something would be very wrong.  But the social sciences have more to offer, they are theoretically sounder and work in them is more rigorous. The result is that social science generally, not just economics, has become a valuable and in some areas maybe even an indispensable tool of legal policy analysis.

But another reason for the spread of quantitative empirical research in the nation’s law school has nothing to do with quality; rather it is simply that it has become so easy to do. Large datasets, canned statistical programs, undergraduate social science training that includes statistics, and the availability of methodologically more knowledgeable colleagues willing to consult means that almost any law professor can do an empirical study if he or she wants. And it is both fun to do empirical research and intellectually rewarding. Traditional doctrinal analysis can produce a new argument, but empirical social science research can produce new knowledge. Discovering something you didn’t know, and which no one else knew, and which seems to matter for some important issue is a kick.

These conditions can be celebrated, but it should also be recognized that they are not all to the good. The ease of doing social science by people minimally trained in it invites poor research. The cookbook application of canned programs to data can mislead if prior steps to ensure the application is appropriate for particular data sets (e.g. investigations of the appropriate functional form for a regression) are not taken. Subtle points like threats posed to by selection bias or how we – or the institution supplying the data – has operationalized different variables may be missed. Even worse is a misapprehension of what social science is about. It is easy to explore data seeking to make a point, and it can be tempting to choose from a larger array of data exactly those variables and that mode of analysis which best makes the point. This not good science, and it will not lead to good law. Instead it can lay down false paths which mislead both scholars and policy makers and take considerable effort to correct, should anyone be interested in doing so. Poor research also discredits our enterprise in the minds of the bulk of our colleagues who consume and critique empirical research but do not do it. We should not be striving to show that something is true, but rather we should be striving to test whether something is likely to be true. We should thus, as I suggested yesterday, strive to generate the best theory-driven hypotheses and then test them rigorously and fairly against their most plausible rival hypotheses. Moreover, while the dangers I mention may be greater with the minimally trained, they are not limited to them. Having a relevant Ph.D. is no guarantee one will do quality work and can enable more sophisticated result finding.

There is also another concern I have. Empirical research is driven by data. Researchers find and create data, but this is often difficult to do. It is expensive; it takes time, and it takes training. Collecting high quality data is hard. It is not a task for the untenured. Yet law professors sometimes do miracles when it comes to collecting valuable data on a shoestring, at least relative to the grant a social scientist would insist on having before beginning a major data collection project. Maybe this is because like Underhill Moore we can enlist young Lloyd Cutlers for a reasonable price, though no one today would have temerity or be permitted by an IRB to assign his students to collect his research data. Law faculty today are learning how to analyze data, but few are learning how to collect it. This can, of course, be self-taught (witness my colleague David Chamber’s wonderful book, Making Father’s Pay and the data he collected to write this), but even this is becoming more difficult as technology plays a larger role in data collection and IRBs set rules about what is and is not allowed when human subjects concerns are implicated. Coding and cleaning data are also important activities that require considerable skill and effort. Why should one do this hard work when large scale precoded data sets, like homicide data coded by county, litigation data coded by case type and district, or Supreme Court votes and the characteristics of judges who cast them are available for free and in forms that can be fed directly into programs for statistical analysis. We must not penalize those whose productivity is slowed by the need to collect, clean and code their own data. Instead the law school world, and in some measure the social science world as well, must come to recognize data collection and preparation as an important scholarly activity which if done well is both a productive use of research time and a potentially major contribution. Gathering quality data and preparing it for analysis is not something anyone can do.

Despite these concerns I am far happier with the situation in law schools today than I was when I started law teaching and am delighted that there are so many people interested in doing or consuming empirical legal analyses. I think empirical legal studies is flourishing as never before because many of the core policy arguments that arise in the law are about whether to attempt to alter or maintain a status quo. This requires appreciating what the status quo is and how it is likely to be altered in particular ways by specific interventions. The law has come to realize that empirical social science is today better able to inform us about the status quo and  the likely impacts of legal changes than the intuition or speculation of even the wisest observer or conclusions drawn from abstract assumption-based models. This does not, however, mean that today’s social science snapshot will be correct; indeed, as I suggested in an earlier blog, policy makers are wise to demand a number of consistent social science snapshots before acting on the assumption that they have the true picture. Moreover, even if today’s social science picture is accurate, it may not be accurate tomorrow because people and conditions change and, in particular, react to changes in policy. This is good news; we empirical researchers will not soon be out of business.

Looking toward the future, I do want to note one concern. Social science in law schools, perhaps for understandable reasons, tends to lag behind the social sciences being done in the disciplines. There is a borrowing or translation process that can be lengthy. This is most easily seen in law school economics since in the law school world that is the longest and most firmly entrenched and respected social science. Game theoretic approaches had come to dominate economic modeling long before they had a substantial presence in the law and economics scholarship, although game theory has a natural fit to many law and economic problems. Behavioral and experimental economics were similarly transforming economics studies years before they began to seriously contribute to law school understandings of economic phenomena. These lags meant that even good law and economics scholars were not doing the best possible economics. Similar observations can be made of scholars whose disciplinary training is in other fields.

What are the modern cutting edge trends in the social sciences that we should now be attending to? Here are four that I draw from my NSF experience. The first is greater attention to space using GIS systems and other mapping and information integration technologies. A second is agent based modeling; specifying parameters of human behavior at the individual or agent level, and through simulations tracing out and exploring the possible underpinning of otherwise impossible to predict outcomes. A third is network analysis, exploring the implications of different links between people and/or organizations for robustness to disruption, coalition formation, information flows and the like. The fourth is using fMRI scans in experimental studies that uncover relationships to brain and behavior. All have obvious connections to problems that concern lawyers. So if you want to create a cutting edge for legal scholarship, forget the analyses that SPSS or STATA enable and move into one of these empirical research areas. Just drop a footnote that you heard it first here.

This is the last of my invited blog. I want again to thank those who maintain the ELS web site and who invited me to be this week’s guest blogger. I also want to thank the surprising number of people who seem to have read my blogs and either publicly or privately have provided me with feedback. It is a pleasure to see empirical legal studies flourishing in law schools in both its quantitative and qualitative forms as never before.

 
CODA

 I have in several of these blogs been free with suggestions and advice on how the analysis of quantitative data should proceed. I hope I haven’t sounded too preachy. But let me mount the pulpit again to make one final suggestion. Namely, and I hope this doesn’t sound too salacious, if you are going to do empirical work be intimate with your data. Converse with it; know its shape, get a feel for how it may be transformed, for what its quirks are, its extremes and its central tendencies. Know where it lives and where it came from. Know how it was born and defects that arose during its birth or development. Don’t become too enamored of it, for better data may be around the corner. Listen to what it tells you, but don’t tell it in advance what you want to hear. Test how sensitive it is to different ways in which it might be approached and try approaching it with different assumptions, if they seem plausible. Do these things, and you’re your data will reward you for it. More importantly, you and we, and the law itself will be better off for your encounter. Also remember, data is not really an “it,” it is a “they;” to show your erudition and not jar your readers write “data are” and not data “is,” and also, by the way, use sunscreen. J J

 

            

Comments

[Richard Lempert]

CODA 2:
The late great Yankees (and later Mets) Manager, Casey Stengal is reported to have once said that he dreamed he died and went to heaven. He hadn't been there very long when the Lord approached him and said he had always wanted to have a baseball team and would Casey be the Manager. Casey looked around and saw all the great players, Babe Ruth, Lou Gherig, Cy Young, Christy Mathewson, Willie Mays, Joe Dimagio, Hank Greenberg, Ty Cobb, etc. etc. and said he would be delighted. The new team finished spring training in the peak of condition. The only problem was that apart from intrasquad games there was no one to play.
Just when Casey was bemoaning this fact the phone rang in the dugout. It was Satan, saying he had just organized a baseball team in Hell and wanted to challenge the heavenly team to a game. Casey was thrilled, but he could not help blurting out, "But you don't have a chance; I've got all the players." "No," said the devil, "You don't have a chance; I have all the umpires."
Empirical legal scholars, take note. The law is the umpire. Don't give up on the analsyis of doctrine.

[michael webster]

This was an interesting piece, but I would have to disagree with the conclusions about the projects suggested -although they are very interesting.

But the real and important work in this field should be to provide empirical support for the rules in the main areas of law, rules which have been largely the product of vicarious strategic reasoning without the benefit of experimental data.

Here are two important areas that this work could be done in.

Criminal Law and the Rules of Evidence - The Rules of Evidence are in part used to exclude statements which are said to be high in prejudicial value, but low in probabtive value. This entire industry has grown without the slighest attempt to verify that the blocked inferences are not made by jurors. In some commonwealth countries, it is actually illegal to obtain these facts. I suspect that it is highly unlikely that the inferences which the rules of evidence block are 1:1 with the inferences that jurors actually make. The social science of framing would provide valuable insight.

2. Tort of Misrepresentation and Misleading Advertising - The elements of this tort include: a material statment and reliance.
Again, what counts after the fact as material and reasonable reliance has been constructed over the centuries by vicarious strategic reasoning without regarding to experimental results. Our disclosure laws, in securities, franchising, and consumer protection all incorporate this highly formal view.

Here is a simple example. The FTC, in the US, regulates the disclosure required for the purchase of a franchise, what material facts must be disclosed by the franchisor before the sale can be completed.

On the first page of the UFOC, uniform franchise offering circular, under the nice FTC logo, the FTC disclaims in large type that it has not checked the facts in the UFOC and encourages the prosepective franchisee to do so, with the help of an attorney or accountant.

Lawyers are surprised to learn that despite this warning, many franchisees thought that the FTC did warrant the facts in the UFOC and that they didn't have an attorney check the facts.

Social scientists would not be surprised that the authority of the FTC logo could outweigh the semantice information in the disclaimer.

Social scientists would not be surprised that near the time of purchase, the prospective franchisee only seeks confirming information and so consults no one, or does so resenting the amount spent as a waste of time.

In conclusion, while the four projects recommended sound interesting, it would be, in my opinion, more helpful to concentrate on the main elements in law and to provide empirical findings on these.