« ELS Ranking Continued: Law Schools as part of the University | Main | Empirical Test of Contractualism in Bankruptcy Theory »

28 September 2006


buy viagra

I am hoping that the current flurry of interest in empirical research among law schools will proceed with careful attention to high standards and will recognize the relevance and importance of both qualitative and quantitative work.

Howard Erlanger

I heartily concur with this opinion from two highly respected empirical researchers. Like JSP and the CSL&S, the Institute for Legal Studies -- established in 1985 at the University of Wisconsin Law School -- developed from a longstanding institutional interest in combining law and social science –- a tradition which dates back at least to 1914, when a Wisconsin professor gave a paper on Eugen Ehrlich’s “Living Law.” Like Edelman and Simon, I am hoping that the current flurry of interest in empirical research among law schools will proceed with careful attention to high standards and will recognize the relevance and importance of both qualitative and quantitative work. Careful work will lead to more tentative conclusions, which will in turn capture fewer headlines. But it will make the best use of what social science has to offer law.

Jonathan Simon

As participants in two of the oldest programs in empirical legal studies, the Center for the Study of Law & Society (est. 1966), and the Jurisprudence and Social Policy program (est. 1978), both at UC Berkeley, we could not be more pleased at the energetic burst of empirical work on law represented in this ranking survey. It has taken alarms going off every two or three decades but it appears that academic law in the US is awakening from its analytic slumber. Nonetheless we believe that the survey methodology was flawed and gives a distorted picture of the academic landscape and more importantly a misleading idea of what empiricism in legal studies represents.

First, no survey is better than its sample and there are real limitations with using Westlaw's JLR database as a measure of productivity in empirical legal studies. Most importantly, the JLR database does not include the top disciplinary social science journals in which many empirical legal scholars publish their empirical articles (including the American Journal of Sociology, the American Sociological Review, the American Economic Review and the American Political Science Review). Scholars with social science training often prefer these journals because they are peer-reviewed. The JLR database not surprisingly reflects the overall market for academic legal source material, inevitably a lagging indicator of any empirical turn in that field.

Second, searching for the term "statistical significance" is a very poor measure of empirical scholarship. The Westlaw search term that was apparently used in this search (“statistic! /1 significan!”) would miss the many empirical articles that use measures of statistical significance but do not use the term ‘statistic’ (or its variants) within one word of ‘significant’ or its variants. It is common to discuss a finding and then to put the significance level in parentheses (e.g. “p<.05”), or simply to write "significantly different from" or even "different from" or any number of other similar phrases. It is also common to report significance levels in tables only, or to report standard errors (so that the reader can calculate the degree of statistical significance), without using the term “statistical significance.” Of course, quantitative analyses of a population rather than a sample would not need to report statistical significance to address sampling error (although some scholars do so anyway).

Third, searching for any term that might indicate empirical scholarship tells us nothing about the quality of that scholarship. As Epstein and King noted in “The Rules of Inference” (69 University of Chicago Law Review 1), empirical legal scholarship varies substantially in quality, and work published in law reviews can suffer from a lack of peer review.

Finally, and this is a much more general problem, the survey assumes that only quantitatively based studies are truly empirical. Quantitative studies (using proper samples and methodology) can provide general descriptive and inferential knowledge, for example about the level of a particular effect from a particular socio-legal dynamic). But qualitative empirical studies using interview, ethnographic, observational, archival or similar methodologies should be viewed as a critical part of the empirical landscape as well because they can offer important insights into the dynamics underlying the effects demonstrated through quantitative work and can be useful in generating hypotheses to be tested using quantitative work. Consider, for example, Robert Ellickson’s study of dispute resolution among Shasta County ranchers and farmers (Order Without Law); Patricia Ewick and Susan Silbey’s empirical studies of legal consciousness (The Common Place of Law); Michael McCann’s analysis of the legal mobilization of rights in the pay equity context (Rights at Work); Catherine Albiston’s empirical analysis of how institutionalized conceptions of work, gender, and disability shape workers’ mobilization of their rights under the Family and Medical Leave Act (39 Law & Society Review 11); Nicholas Pedriana’s analysis of legal framing processes in the women’s movement (111 American Journal of Sociology 1718); and numerous other studies that enrich our understanding of legal processes due to careful analysis of qualitative data. (We disagree with an earlier commentator who wrote that practically all legal scholarship includes qualitative empirical analysis – it is important to distinguish work that uses qualitative data and proper empirical methodology from traditional doctrinal or normative analyses, which often do not involve any empirical analysis).

Lauren Edelman, Director, Center for the Study of Law & Society, UC Berkeley

Jonathan Simon, Associate Dean for Jurisprudence and Social Policy, Boalt Hall, School of Law, UC Berkeley

Christopher Zorn

Hm. I guess my concern would be that -- on the other, hyper-quantitative end of the spectrum -- fewer and fewer well-trained social scientists are buying into the whole "statistical significance" / null-hypothesis-testing paradigm of inference. For a good explanation of why that is, look here:


That means that the measure will increasingly miss instances where people are, in fact doing statistical/empirical work but not using the terminology.

kal raustiala

it's an empirical question whether "els" refers only to quant work, but in my view it shouldn't. and i don't think it is quite accurate to say that nearly all legal scholarship includes qualitative empirical analysis. much is simply textual, doctrinal, or otherwise concerned with particular legal rules (themselves empirical phenomena, but of a very special sort) or is theoretical in nature. actual qualitative study of the effects of legal rules, for instance, isnt all that common in my experience. and what does exist is rarely methodologically self-conscious.

Tracey George

Kal clearly is correct that the term "empirical" includes both qualitative and quantitative work. However, "empirical legal scholarship" generally refers to work with quantitative analysis as a central component, including both experimental and observational data. When law professors talk about the rise of ELS, they usually are referring to this quantitative work only. Almost all legal scholarship includes qualitative empirical analysis.

kal raustiala

i'm glad to see my colleagues score well on this particular measure but the fact remains that empirical scholarship and quantitative analysis are not the same thing. in the social sciences we generally consider qualitative work to be equally empirical (though we argue, at least in political science, where i come from, about whether it is as rigorous). so to equate empiricism with regressions is a bit odd.

Royce Barondes

I have some suspicion that the false positives are larger than suggested by the reference, "Or the article may describe the concept in more meaningful terms for a lay audience and to avoid confusion with practical significance." If an article merely includes a footnote identifying another's work followed by a parenthetical such as "finding a statistically significant relationship", the article is classified as empirical. A brief scan in "kwic" view of a few results of this search suggests the majority of articles identified by this search may well not be empirical.

frank cross

Chris should get extra points for writing with Jeff.

Tracey George

I ran separate searches for each school, listing each faculty member as a possible author. Thus, I don't have information on all hits.
What percentage of those hits are at top 50 schools? I can only give you an estimate. The top 50 schools have a total of 1503 citations. However, these are not unique citations due to cross-school collaborations. Thus, Jeff Rachlinski (Cornell) and Chris Guthrie (Vanderbilt) write an article discussing statistical significance, and both schools receive credit for one citation. (Collaboration within a school, however, does not result in multiple credit to that school for one article.) Collaboration is more common in ELS than in other work. Thus, I wouldn't be surprised if top-50 law schools accounted for only 20% of all hits.

Jason Czarnezki

I, for the most part, like the measure. I just ran the search on Westlaw -- STATISTIC! /1 SIGNIFICAN! & DA(AFT 1/31/1995 & BEF 9/28/2006 -- and received 6845 hits. Did you code all of these for school affliation and whether the author is part of the research faculty"? If so, do you have the rankings for all the schools that show up? Do any lower ranked schools (based on U.S. News) perform comparatively better under this measure?

The comments to this entry are closed.


July 2024

Sun Mon Tue Wed Thu Fri Sat
  1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30 31      

Site Meter

Creative Commons License

  • Creative Commons License
Blog powered by Typepad