Empirical Legal Studies

Survey Research

I am currently in the process of designing a survey on financial and billing information for the Indiana State Bar Association (ISBA).  In preparation for this task, I have read and reviewed lots of surveys by bar associations in other jurisdictions.  There are a surprising number, often conducted by local academics (I blogged about a really terrific survey of Pittsburgh lawyers here). 

What I lack, however, is a good reference guide for interpreting survey data, particularly a methods book or article that discusses how to check for a representative sample and various weighting techniques for either correcting sample bias or imputing missing values based on known characteristics of the population (information I have because the survey is sponsored by the ISBA). 

Any suggestions in the comments or by email would be greatly appreciated.

Closed Med Mal Insurance Claims

The folks at BJS recently released an interesting report on data from seven states that require med mal insurance carriers to report information on closed claims. The report, Medical Malpractice Insurance Claims in Seven States, 2000 - 2004, focuses on:

"... medical malpractice claims that were closed in Florida, Illinois, Maine, Massachusetts, Missouri, Nevada, and Texas from 2000 to 2004. In these states medical malpractice insurance providers are required by law to submit information on closed medical malpractice claims. These claims are typically submitted to the Department of Financial and Insurance Regulation, which provided the data to BJS. The report provides an in-depth analysis of medical malpractice claims, including the number of claims closed with or without payouts, types of medical providers named in claims, facilities where alleged injury occurred, severity of alleged injury, court disposition of claims, amounts paid to compensate claimants, costs of defense counsel, and claim processing time."

Vault Top 25 Underrated Schools: What Does It Mean?

Vault, Inc, which is a publisher that collects industry information on various professions, recently released its listing of the Top 25 Most Underrated Law Schools.   (Hat tip: Paul Caron & Volokh.)  The findings are derived from a survey of  512 law firm recruiting managers, hiring partners, and corporate counsel, who were asked "to name law schools that, based on their experience as hiring managers, are underrated."

As I read the list, I wondered, what is the practical implication of a law school being underrated by the people who make hiring decisions for entry level lawyers?  Presumably, it means that graduates of certain law schools tend to perform better than their school's U.S. News ranking would suggest; thus, legal employers are more likely to hire them. 

If this is true, what is the source of superior performance? Here are two possibilities:

1. Stronger Students.   Some schools may enroll a stronger student body than their rank might suggest.
2. Better Education.  Some law schools may equip graduates with more or better skills than other schools of comparable rank

Based on some preliminary statistical analysis, there is fairly clear quantitative evidence for the first hypothesis.  There is also some qualitative evidence for the second--enough to warrant some additional research. 

These research questions are really important to the ranking debate because their answers could reveal a mechanism by which some students will discount USN.  If an "underrated" school offers a better entree to coveted legal employers, why go to the higher ranked school, especially if the underrated school costs significantly less?  Law schools that understand these dynamics are in a better overall competitive position.

After the jump, I elaborate on these findings, including methodology.

A Serious Debate on Empiricism and Theory in Bankruptcy

Last April, we posted the abstract of Robert Rasmussen's (Vanderbilt) working paper, "Empirically Bankrupt," which critiques three empirical papers in the bankruptcy area.  The abstract concluded that "[f]or empirical work to be credited, at a minimum, it has to look in the right place, ask the right question and draw the right inferences. When empirical work fails to cross this threshold, it conclusions must be rejected."

One of the articles being critiqued was Elizabeth Warren's (Harvard) & Jay Lawrence Westbrook's (Texas) recent study, Contracting Out of Bankruptcy: An Empirical Intervention, 118 Harv. L. Rev. 1197 (2005).  A few months ago, Warren & Westbrook posted a rejoinder, "The Dialogue Between Theoretical and Empirical Scholarship," that not only addresses Rasmussen's criticism, but also tries to articulate an optimal relationship between theory and empiricism.   The entire exchange between Rasmussen and Warren & Westbrook is worth reading. Here is the abstract of the rejoinder essay:

In this essay we offer brief reflections on the best process for critiquing empirical work in law and sustaining an engagement between theoretical and empirical approaches. We emphasize the importance of theoretical work in helping to shape the scholarly agenda, but we urge that theory should be more closely tied to fact. We illustrate our argument by responding to a recent critique of our own empirical work by Professor Rasmussen. His principal claim is that our work should be discounted because we reported on all business bankruptcies, both those of entrepreneurs and those in corporate form. In response, we reanalyze our data, separating the individuals from the corporations; in every case the re-analyzed data support the conclusions of our original paper to the same extent or more strongly.  Similarly, his other claims about our work are shown to be incorrect.

Judicial Decisions as Data Points & Precedent

Over at the Social Science Statistics Blog Jim Greiner helpfully reminds us about the salience of precedent for those endeavoring to model court decisions. Jim notes (here):

"It's tempting to think that one can code appellate decisions or judicial opinions pursuant to some neutral criteria, then look for trends, tease out inferences of causation, etc. Here's a note of caution: they're not i.i.d. They're probably not i.i.d. given X (whatever X is). Precedent matters."

Judicial Ideology & Case Selection Effects

Ahmed Taha (Wake Forest) circulated an interesting paper, Judges' Political Orientations and the Selection of Disputes for Litigation," focusing on the influence of judges' ideology on settlement behavior. Taha's paper explores various case types and assesses whether (perceptions of) judge ideology influences which cases settle and which go forward. The study finds that:

"... at least for some types of cases, judges' political orientations have significant effects on the number of cases filed. Thus, the political orientations of even trial court judges have important effects on the outcomes of the federal justice system: fewer potential plaintiffs seek relief in federal courts if they believe the judges are less likely to be sympathetic to their cases. This finding also helps explain the surprising findings of other researchers that a federal district judge's political orientation generally does not affect the probability that a case ends in a judgment for the plaintiff or a settlement. This paper's findings suggest that those results are based on a selection bias: because parties believe that judges' political orientations matter, they are likely settling cases on more favorable terms for the plaintiffs if the cases are assigned to Democratic judges rather than to Republican judges."

Justices Who Change Over Time

This week, the Colloquy published two pieces as it begins a series of responses to an article  forthcoming in the print Northwestern Law Review by Profs. Lee Epstein, Andrew D. Martin, Kevin M. Quinn, and Jeffrey A. Segal.  In it, Epstein et al. perform an empirical analysis of Supreme Court Justices' voting patterns and find that almost all Justices exhibit some degree of shifting ideological preferences during their time on the Court. 

Colloquy posts the introduction to the full piece by Epstein et al (see here), and follows with a reflective piece by Linda Greenhouse, the Supreme Court correspondent for the New York Times and author of Becoming Justice Blackmun (see here).  In her piece, Ms. Greenhouse draws on her time covering the Court and her study of Justice Blackmun, the exemplar of shifting Justice ideology, to provide anecdotal support for the conclusions of Epstein et al. 

Judges Use Law Reviews "Like Drunkards Use Lampposts: More For Support Than for Illumination"

Paul Caron  links to a devastating (and devastatingly funny) article in the New York Lawyer, entitled "Federal Judges Discuss Law Reviews Usefulness."   Paul excerpted some the sharpest lines:

Kicking off a recent friendly-but-frank discussion about the relevance of contemporary law review articles, Dean David Rudenstine of the Benjamin N. Cardozo School of Law offered one description: "useless blather puffed up with self-indulgence" ...

Judge Sack returned to the not-so-funny problem under consideration: today's highly theoretical articles are largely ignored and seldom cited by judges, a dramatic turnabout from a generation ago when the mostly practical content of law reviews was a significant element of judicial decision-making. Judge Sack was sorry to say that the bench now uses law reviews "like drunkards use lampposts, more for support than for illumination." ...

"If the academy wants to change the world, it must decide if it wants to be a part of the world." ...

Although Judge Sotomayor agreed that brainy law professors should not trouble themselves by contemplating reactions from the bench to their writings, she leveled a sharp gaze at the Cardozo Law faculty and declared, "If you think that judges are not as capable of creative thought as you are, I beg to differ." She added, "My question to academics: do you really think you're serving some function to someone?"

Wow, that is some really straight talk.  For the record, law reviews had a sketchy record 40 years ago, see Lowell J. Noteboom & Timothy B. Walker, The Law Review–Is it meeting the needs of the Legal Community?, 44 Denver L. J. 426 (1967).

Revisiting State Supreme Courts

A recent paper by Herbert Kritzer (Wisc.--poli sci) et al., The Business of State Supreme Courts, Revisited, sets out to update Kagan et al.'s earlier study of the docket composition of state supreme courts. I am partial to such undertakings (updating, revisiting, and/or replicating important past work) and welcome attention to the comparatively understudied state courts. Kritzer and his colleagues find:

"... many of the patterns of change described by Kagan et al. continued through the 20th century: debt and real property continued to decline and criminal continued to increase. However, other patterns of change either reversed or halted. Specifically, neither torts nor family cases have continued to increase; torts have stabilized and family cases, rather than increasing, have declined. The most surprising shift is the sharp increase in "other contract", which had no particular pattern in the earlier data, but which represented five percent or less of the courts' business; in the 1990s, "other contracts" had grown to a level approaching that of public law, and exceeding real property and family and estate cases."

LSA Preliminary Program Now Available...

...see here.

Splitting the Ninth

Last week, Justices Kennedy and Thomas visited Congress to present the annual budget request and, while there, discussed the need (as they see it) to split the Ninth.  (Hat tip to How Appealing Extra.)  They are not, of course, the first to discuss this, but I thought I'd use it as an opportunity to think about what ELS can provide to an important (and sometimes heated) policy question. Very often Congress (or state legislatures) make policy decisions without "good data."  (States' choices of selection systems, for example, rarely seem to be informed by the ramifications of selection method on decision making or legitimacy, etc.)  So, to start a discussion of what a split of the Ninth would accomplish v. what harm would it do, I call to your attention work by Kevin Scott of the Congressional Research Service (and I'm hoping others might call attention to other empirical work on this question as well).  Here's his abstract:

Much attention has been focused on the Ninth Circuit Court of Appeals. To some degree, the attention has been self-created, with high profile decisions on three-strikes laws, punch-card ballots, and the constitutionality of the Pledge of Allegiance generating substantial media coverage and reaction from elected officials. This attention mirrors an ongoing debate about the wisdom of splitting the Ninth Circuit into two smaller circuits. Arguments about splitting the Circuit center on the source of the Circuit’s high reversal rate. There are two competing explanations for that reversal rate: the judges of the Ninth Circuit are more ideologically distant from the Supreme Court than judges on other Circuits, or the size of the Ninth Circuit prevents the circuit majority from correcting “wrong” decisions by panels, a problem augmented by the Ninth Circuit’s reliance on the limited en banc procedure.

This paper sorts out the cause of the Ninth Circuit’s reversal rate by looking at the Court’s relationship with the Supreme Court over the past twenty years. By looking at merits reversals, including unanimous reversals, and attempting a broader assessment of the Ninth Circuit’s status vis-à-vis the other circuits and the Supreme Court, I ultimately argue that both size (though indirectly) and ideological orientation influence the Ninth Circuit’s high reversal rate. These findings have broad implications for how we model the behavior of court of appeals judges and their relationship with the Supreme Court.

For the paper, click here.

A listserv for junior ELS scholars...

Recent guest blogger Carolyn Shapiro and her colleague at Chicago-Kent, Bernadette Atuahene, are proposing a new listserv for junior scholars interested in empirical legal scholarship.  The purpose of the listserv would be to provide a forum for folks to seek advice on methodological, statistical, and other types of questions.  The idea is modeled, very generally, on the successful Chicago-Area Junior Faculty Workshop, which brings together junior scholars from the Chicago area to present and discuss each other's work. Here, the idea would be to provide a nonthreatening environment for people to ask questions and seek out resources.  (In this context, junior means non-tenured, whether pre-tenure or not tenure-track.)  If you are a junior faculty member interested in this, please e-mail [email protected] with "ELSlistserv" in the subject line.  If there's sufficient interest, we'll proceed.

Co-Authorship

One article I have been meaning to blog about is this profile of economist Kevin Murphy in the Nov/Dec 2006 issue of University of Chicago Magazine.  Murphy has an impressive list of accomplishments, including the John Bates Clark Medal of the American Economic Association, which is given once every two years  for the most outstanding American economist under age 40 (1997), and a MacArthur Foundation "genius grant" (2005).

Murphy is quite a character, and the article provides some fascinating details on his career history.  But this passage on co-authorship really caught my eye:

"[Regarding co-authorship,] I think most people get a better product that way,” he explains. “You pool insights and eliminate oversights. First-rate coauthors make working on projects more fun and rewarding. Plus, writing is a pain.” Since earning his PhD and joining the faculty in 1986, he has written every one of his 60-plus published papers with a former teacher, a close colleague, or students. ...

As a shy but affable 25-year-old grad student, Murphy confessed to [his advisor] Topel that his goal at that stage in life was “to become the world’s best coauthor.” By the time he was 35, he was there.

Wow, that is quite a testament to collaboration.  To the right is a picture of Murphy, who is an avid woodworker, in his workshop at home.   Apparently, it is impossible to locate a picture of Murphy without his signature baseball cap. 

Replication: A Difficult Endeavor Made Even More So

If replication was not already difficult in many instances, a recent post at the Social Science Statistics Blog reveals that, for studies using certain finance data (specifically, data on stock analyst recommendations), the degree of difficulty is materially higher than one might imagine.  Evidently, certain data sources (notably in the finance sector) resemble "ever-shifting moving targets."

Multidimensional Scaling - Blumenthal #5

Another useful methodology often used in psychology -- less in social, more in cognitive -- is one alluded to here before, "multidimensional scaling" (MDS).  MDS has only been used in a small number of legal analyses, but can be quite helpful in the right context.

As I describe in a forthcoming paper, "MDS is a procedure that helps researchers uncover 'hidden structures' in data by graphically plotting respondents' perceptions of perceived similarities (or dissimilarities) among various stimuli.  When these stimuli are located on a plot based on such perceptions, underlying dimensions that respondents may have used (consciously or not) can be inferred."

So, for instance, a bunch of stimuli are identified -- crimes, Supreme Court opinions -- and every pairwise combination is created.  Each pair is rated on how similar or close they are (though the researcher usually does not articulate the scale to be used, just asks "how similar"?).  Those similarity ratings are then plotted to give the best fit to the data, yielding a "map" of the ratings that also reflects the perceived structural relationship of all those stimuli.  Finally, by examining the map, one might be able to identify the dimensions that, implicitly or explicitly, underlay subjects' ratings (and also, interestingly, look at individual differences, e.g., do men and women or Republicans and Democrats come up with different maps or weight those dimensions differently?).

As a specific, somewhat prosaic example: subjects might be asked to rate the similarity of countries in the world, and a plot made of the countries based on those similarity ratings.  The resulting map might look like a map of the world, which could suggest that subjects were, implicitly or not, using dimensions involving distance, in particular, dimensions of north/south and east/west.  But the plot might line up the countries on a single axis going from largest to smallest, or richest to poorest, or most to least economically developed, suggesting that subjects used that single dimension in their ratings.  Inferring which dimensions are important to respondents helps understand the way they see the world (literally, here), and what factors are important to them.

In the legal context, I used it in that paper to map people's perceptions of the seriousness of crimes.  MDS is broadly useful for looking at lay intuitions about legal institutions, punishments, or cases; or to assess whether different punishments may have been proportionate; or any context in which (a) similarities among stimuli may be interesting or (b) we are interested in understanding what the basis might be for people's judgments or perceptions.

A classic book on MDS is by Kruskal and Wish, from the 1970's.

Conducting Empirical Legal Scholarship Workshop

On Monday, May 21, 2007 through Wednesday, May 23, 2007 Lee Epstein and Andrew Martin will be teaching their annual Conducting Empirical Legal Scholarship workshop. This workshop will be held in St. Louis, and is co-sponsored by Northwestern University and the Center for Empirical Research in the Law at Washington University. Additional information about the workshop can be found at:

http://cerl.wustl.edu/training/cels.php

The Conducting Empirical Legal Scholarship workshop is for law school and social science faculty interested in learning about empirical research. The instructors provide the formal training necessary to design, con duct, and assess empirical studies, and to use statistical software (Stata) to analyze and manage data. Participants need no background or knowledge of statistics to enroll in the workshop. Topics to be covered include research design, sampling, measurement, descriptive statistics, inferential statistics, and linear regression.

Please visit the website for all logistical details. Feel free to contact Andrew Martin ([email protected]) with substantive questions, or Karma Jenkins ([email protected]) with logistical questions.

Insights from the Methodology of Psychology (Blumenthal #4)

Picking up on the methodological contribution I think psychology can make to ELS, it seems as though psychology emphasizes experimental work to a greater degree than current ELS. Discussing the excellent methodology workshop that Lee Epstein and Andrew Martin run, someone recently joked that it covers 95% of all the methodology ELS scholars need to know. I’d quibble with that if, as I understand (without having attended), there’s little coverage of experimental design and analysis (e.g., analyses of variance or ANOVA’s). To be very clear, all I mean by that is I think the experimental approach could be a larger proportion of ELS work than it currently is.

Experimental research, of course, can contribute greatly to our understanding of a number of topics and phenomena (jury understanding of sentencing instructions; eyewitness accuracy; racial discrimination by judges and jurors; effects of policy interventions; different influence of legal procedures both in and out of the courtroom, etc., etc.). Researchers can manipulate specific variables and test their causal effect, tweaking particular aspects to really parse out specific effects that may be more precise than multiple regression. In some instances – when a researcher is working with existing databases – experiments can be less constrained by data limitations, can be more proactive in data collection.

Among the downsides, of course, is the perception that experiments aren’t “externally valid” – they don’t sufficiently reflect the real world that we can profitably generalize. Sometimes, true. But they sometimes do, and sometimes can be conducted in real-world settings. Other times, finding an effect in the lab suggests it might be even stronger in the real world: the example often given is with comprehension of capital sentencing instructions – when highly educated mock juror subjects from “elite universities” don’t understand them, how much more of a concern might it be for less educated jurors faced with such instructions in the real world? In yet other contexts, again with mock juries, differences between undergraduate samples and community member samples are substantially less than critics have suggested (Bornstein, 23 Law & Hum. Behav. 75 (1999)).

I don't want to oversell the experimental approach, but it could certainly be another arrow in the ELS quiver.  So to speak.

Can More Money Reduce the Rate of Associate Attrition?

A recently published story in The American Lawyer asks the salient question, "Is Raising Salaries the Best Way to Retain Associates?"   It is always great to get an unsolicited $20,000 raise, but what if your real gripe is that you work too many hours and your physical health and family life are falling apart?  Or that there is no communication on partnership chances, making sticking around too risky?  It is possible that managing partners and executive committees do not know the actual price points--monetary and non-monetary--of their associates.

Some of the findings presented in my essay with David Zaring, "Young Associates in Trouble," suggest that these law firms may, in fact, be throwing good money after bad.  In the table below, which is based on data from the Am Law Midlevel Associate Survey (3rd, 4th, and 5th year lawyers), the dependent variable is likelihood on a 1-5 scale that the associate will remain with the firm during the next two years. 

The regression results tell a very straightforward and intuitive story:

• Carrots.  Midlevels are more likely to stick around if (a) they do interesting work or work on quality projects (which I suspect means career enhancing), (b) are given cues on partnership prospects, or (c) the firm has a large nonequity tier (this study explains why this is attractive to non-rainmaking lawyers).
• Sticks.  Midlevels are less likely to stick around when the hours are long and the environment is less family friendly.
• Irrelevant.  Controlling for all of these facts, higher salaries and bonuses appear to have no statistically relationship to Midlevel's self-reported likelihood of staying with the firm.

U.S. Supreme Court Justices Database

The U.S. Supreme Court Justices Database, compiled by Lee Epstein, Thomas G. Walker, Nancy Staudt, Scott Hendrickson, and Jason Roberts is now available for public use. According to the principal investigators, “This is a multi-user, public database containing a wealth of information on individuals nominated (whether confirmed or not) to the U.S. Supreme Court. Specifically, the database houses 263 variables, falling roughly into five categories: identifiers, background characteristics and personal attributes, nomination and confirmation, service on the Court, and departures from the bench.”

The data are archived on the website of Lee Epstein at Northwestern University School of Law at  http://epstein.law.northwestern.edu/research/justicesdata.html.

Hat tip: Kirk Randazzo

Blumenthal #3 - More on Psychology

As a follow-up, I would point out some excellent psychology-and-law resources (sorry; now that I teach it at a law school, it's "law-and-psychology" - sort of emphasizes the point, doesn't it).

First is the American Psychology/Law Society, founded in the late 1960's; with annual conferences highlighting the most current and topical research in legal psychology.  The website has links to various newsletters of the Society, which include conference programs in order to see who is doing what sort of empirical research.

Second is the Society-sponsored, peer-reviewed journal, Law and Human Behavior, published by Springer.  Publishing top-quality empirical research, LHB is consistently one of the most highly-cited non-law-review journals.  The topic areas are often limited to the "traditional" psych-and-law topics I've mentioned before, though - forensics, eyewitness, juries - and I am sure that its editors would welcome a broad range of submission topics of the sort that ELS people here investigate.

Both APLS and LHB are good resources for finding primary research and empirical, interdisciplinary collaborators (JD/PhD's are not uncommon in APLS) - IMHO, both can help as the ELS field continues to develop.

Blumenthal # 2 - A Plea for Psychology (with apologies to John Austin)

A Plea for Psychology (with apologies to J.L. Austin)

So I've gotten off to a slow start, but I'll try to make some contribution now.

As I mentioned, I wonder if psychology as a discipline has gotten short shrift from ELS. As a historical matter I think so; research in legal psychology has been conducted, off and on, for about 100 years. But until recently the legal system has been skeptical, suspicious, at times hostile to psychology, and there’s not much discussion of psychological research in modern ELS – with exceptions, much of the work stems from economics or political science.

There are historical reasons for this skepticism. The conventional story is that psychology and law as a discipline was founded around 1908, with the publication of Hugo Munsterberg’s On the Witness Stand, documenting his efforts to persuade the legal system that his empirical work in criminal justice – eyewitness research, memory, perception, lie-detection, etc. – was relevant and important. (Another interesting contribution, written in a more scholarly vein, was a book by G.F. Arnold, a British legal scholar, Psychology Applied to Legal Evidence (1906)).

Munsterberg's book was pugnacious and had a lot of 'tude -- probably too much ("The time for Applied Psychology is surely near. . . .  The lawyer alone is obdurate;" "The lawyer and the judge and the juryman are sure that they do not need the experimental psychologist. . . .  They go on thinking that their legal instinct and their common sense supplies them with all that is needed and somewhat more.")  Hugo's book prompted a now-classic response article by John Henry Wigmore, in which Wigmore staged a mock lawsuit accusing Munsterberg of libeling the legal profession.  Both tongue-in-cheek and vitriolic, Wigmore's cross-examination devastated poor Hugo, and as a result, one commentator has said, for years American psychologists "left the law rather severely alone."  Last week a wonderful conference was staged at John Jay College of Criminal Justice, though, celebrating OTWS's centenary: Off the Witness Stand: Using Psychology in the Practice of Justice."  With a keynote speech by Janet Reno, and talks by many top researchers in in psychology discussing the state of psycholegal research today--experts in eyewitness testimony, memory, jury decision-making, forensic psychology, coerced confessions, and more--it was geared toward both scholars and practitioners, and was a good state-of-the-discipline weekend.  (One concern, as I hope to discuss in another post, is the impression that criminal justice is all there is for psychology to look at.)

And there are methodological / institutional reasons for the skepticism.  As I and many others have noted, experimental psychology takes a different approach to finding answers: objective rather than adversarial, nomothetic rather than idiographic, probabilistic rather than certain, descriptive rather than prescriptive.

But this skepticism is unfortunate, and has led to a lack of communication between the disciplines of law and psychology.  I think ELS suffers from that lack, in at least two ways.  The first is simply mutual unfamiliarity with the other field’s research.  This is unfortunate historically (for instance, the explosion of heuristics-and-biases research over the last 10 years or so, which builds on psychologists’ work that has been around for 30 years and appears in any introductory cognitive psychology textbook), but also in terms of keeping up with empirical developments in psychology (and other social sciences too, of course).

*

The second, which I'll turn to next post, is methodological. . . .

*

For those who are still here, thanks!

Extending ELS' Reach: From the Ivory Tower to the Real World

The folks at Deliberations, a blog devoted to juries and jury selection, highlight a recent real world application of empirical legal scholarship.  Jason previously noted "A Study of Juror Expectations and Demands Concerning Scientific Evidence: Does the 'CSI Effect' Exist?" and a recent post at Deliberations illustrates its relevance to the Libby trial.

Blumenthal Post #1

I appreciate the invitation to participate in the ELS blog.  As Jason mentioned, my empirical background is in psychology, so I’ll try to highlight a number of insights from work in that field that inform empirical – and experimental – legal studies, as well as point to some interesting ongoing work in psychology and law.  I’ll also mention a few methodological issues I think get neglected sometimes in our ELS discussion, hoping to expand the empirical approach we take to such study. And if I haven’t yet bored everyone by then, I’ll try to come up with other ELS bits as well.  Thanks again!

What are the Top Law Reviews?

Al Brophy (Alabama) has written "Law [Review]'s Empire: The Assessment of Law Reviews and Trends  in Legal Scholarship".  The Abstract:

"Recent research details the close connections between a school's US News rankings and citations to a school's main law journal. This brief essay builds on that research. Drawing upon John Doyle's database at Washington and Lee Law School, it looks to the 100 most-cited secondary student-edited law journals, with the goal of seeing the connections between well-cited secondary journals and school ranking. A final table provides a ranking of the most-cited secondary journals."

Guest Blogger: Jeremy A. Blumenthal

This week's guest blogger is Jeremy Blumenthal, Assistant Professor of Law at Syracuse University College of Law.   He earned his A.B., A.M., and Ph. D. in Social Psychology from Harvard University. He earned his Juris Doctor and graduated cum laude from the University of Pennsylvania Law School, where he was Editor-in-Chief of the University of Pennsylvania Journal of Constitutional Law and received several writing prizes. Most recently, he was a Faculty Fellow at Seton Hall Law School, teaching 1L Property and Law and the Social Sciences.  Previously, he clerked for the Honorable Maryanne Trump Barry of the U.S. Court of Appeals for the Third Circuit, and was an Associate for Latham & Watkins, LLP, in the Newark, NJ office.

Professor Blumenthal has published widely on topics in law and psychology, including demeanor evidence, the reasonable woman standard, victim impact statements, and more generally on the role of social science in assisting the legal system. He has also published in the areas of evidence and comparative law. His most recent publications focus on the role of emotions in the legal system; current research projects address topics in capital punishment, empirical work in law and psychology, and topics in property theory.

Does the U.S. Constitution Need an Equal Rights Amendment?

I just received a reprint of Baldez, Epstein and Martin's Does the U.S. Constitution Need an Equal Rights Amendment? (click on title for link to full text available on the Journal of Legal Studies website).  The Abstract:

For over 3 decades, those engaged in the battle over the Equal Rights Amendment (ERA), along with many scholarly commentators, have argued that ratification of the amendment will lead U.S. courts (1) to elevate the standard of law they now use to adjudicate claims of sex discrimination, which, in turn, could lead them (2) to find in favor of parties claiming a denial of their rights. We investigate both possibilities via an examination of constitutional sex discrimination litigation in the 50 states—over a third of which have adopted ERAs. Employing methods especially developed for this investigation, we find no direct effect of the ERA on case outcomes. But we do identify an indirect effect: the presence of an ERA significantly increases the likelihood of a court applying a higher standard of law, which in turn significantly increases the likelihood of a decision favoring the equality claim.

Instrumental Variables versus Matching Techniques

A few weeks ago, I posted on the Advanced Empirical Methods Workshop at Washington University.  I briefly mentioned one of the presenter's comments that non-parametric matching is a better way to deal with endogeneity than instrumental variables (IV) in a linear model.

That post prompted an email from one of the leading econometricians in the legal academy, who noted that the pros and cons of IV versus matching was a matter of open debate in which the two camps were roughly divided by discipline--political science favoring matching and economists relying on IV.  [For the record, my correspondent claimed that both techniques had their strengths and their relative superiority depended upon the research question and the available data.]

The basic problem that both techniques are grappling with is accurate causal inference from observational data.  In a controlled experiment in which subjects are randomized and divided into treated and untreated groups, we can be relatively confident that subsequent differences (Y) among the two groups are attributable to the treatment (X).  But many social phenomena are not amenable to controlled experiments.    Thus, it is difficult to ascertain whether some unobserved or unmeasurable attribute is really driving our results, thus biasing our parameter estimates for X. 

Matching techniques attempt to achieve the same effect as experimental randomization by "balancing" a sample on all observable attributes.  [Ed Morrison (Columbia Law) blogged on this technique a couple of months ago.] The key assumption is that all unknown/omitted attributes are similar within the matched sample.  In contrast, when there is a good theoretical basis to believe that a particular variable (X) is correlated with a missing/unobservable variable (Z), IV attempts to find an instrumental variable that is correlated with X but uncorrelated with Z.  By substituting the IV for X, the potential endogenous relationship is eliminated.

Critics of IV claim that good instruments are very difficult (or impossible) to find.  But, as my econometric colleague pointed out, occasionally some unusual event occurs that creates an natural experiment--and this "shock" can supply the ideal instrumental variables. 

All of the above is a long way of introducing this terrific article from the Wall Street Journal (subscription required) on how economists have used natural experiments and instrumental variables to make great headway in explaining the causal mechanisms of a wide array of social phenomena  (Hat tip to Tom Ulen at the Law & Economics Blog). 

Under the fold is a highly informative WSJ chart [click to enlarge] summarizing some of the leading IV studies.   

Numbers in the Classroom

It's been fun guest blogging this week, but this is likely my last post.  I'm going to wade into the ongoing discussion here and elsewhere on what kinds of statistical and social science background law students should get (or be offered) and lawyers need.  The easy answer to those questions, not likely to be controversial here, is more than they get!  Perhaps most useful to the students are those classes that focus on the types of numeracy and related skills that lawyers need when dealing with expert witnesses. 

But I don't think that teaching about such matters should be restricted to such classes or to classes covering, for example, empirical studies of judicial decisionmaking.  In my employment law class, for example, I cover the law governing the use of polygraph tests and other kinds of screening tests by employers.  While I'm at it, however, I walk the students through a (very simple) set of exercises aimed at helping them understand the statistics of screening tests.  What does it mean to say that a test is 90% accurate, or that it has a specificity of 95% but a sensitivity of only 70%?  Why might you want to know something about the prevalence or expected prevalence of whatever it is you are screening for?  How do you interpret a "positive" result?  Invariably, most of my students find this discussion extremely intimidating and even panic-inducing.  ("I didn't realize we'd have to do math in this class!" or "Is this going to be on the final?" are the two most common responses.)  Yet an employment lawyer advising a client on whether or not it is worth the legal hassles or potential hassles associated with using and relying on different types of screening tests simply cannot give competent advice without knowing how to think about these questions -- or at least without knowing that those questions exist and need to be answered.

I'm sure most readers of this blog wouldn't disagree and are likely to find ways to include numeracy if they teach doctrinal classes. I'm also  sure many of our colleagues who do not consider themselves ELS devotees do so as well.  It would be interesting to know how much this happens or, to the extent it doesn't, do law professors feel that more would be helpful but may not themselves feel competent in presenting the information.  I believe that statistical and social science information closely related to the doctrinal subjects already taught may be the easiest to "sell" to faculty and students alike -- certainly easier sells than, say, empirical studies of judicial decisionmaking or general statistics classes that are not directly tied to application in they study of doctrine or the practice of law.

How Can Lawyers Find Useable Empirical Info?

Our readers might be interested in a post from Deliberations, a blog devoted to juries and jury selection.  The post is entitled "Which Way to Research?", and discusses the how hard it is for lawyers to find empirical information in a form they can understand and use.

The "CSI Effect" on Jurors

Professors at Eastern Michigan University have written "A Study of Juror Expectations and Demands Concerning Scientific Evidence: Does the 'CSI Effect' Exist?".  The Abstract:

Many prosecutors, judges and journalists have claimed that watching television shows like CSI have caused jurors to wrongfully acquit guilty defendants when no scientific evidence is presented. This is the first empirical study designed to investigate whether the "CSI effect" exists. This survey of 1027 persons called for jury duty in a State court looked at jurors' television viewing habits, their expectations that the prosecutor would produce scientific evidence, and whether they would demand scientific evidence as a condition of a guilty verdict. While the study did find significant expectations and demands for scientific evidence, there was little or no indication of a link between those preconceptions and watching particular television shows. The authors suggest that to the extent that jurors have significant expectations and demands for scientific evidence, it may have more to do with a broader "tech effect" in our popular culture rather than any particular "CSI effect." At the same time, this article contends that any such increased expectations and demands are legitimate and constitutionally based reflections in jurors of changes in our popular culture, and that the criminal justice system must adapt to accommodate jurors' expectations and demands for scientific evidence.

Just the Facts?

Thanks to everyone who has commented on my posts so far.  The responses to my first post, on whether judges accurately report the facts, were extremely helpful.  I think that Sara Benesh may have put her finger on one of the real challenges of doing this kind of work – deciding what facts should have been reported. In situations where the Supreme Court has made it clear that certain types of facts must be taken into account, whether by incorporating those facts explicitly into a legal test or standard or by doing so implicitly by relying on such facts (or their absence) repeatedly, my hypothesis would be that the trial and appellate courts do a pretty good job of accurately reporting them – as Sara seems to have found. 

But my interest in this thorny question arises from a different context.  In employment discrimination cases, courts often must decide whether to grant summary judgment.  So they have to evaluate whether a reasonable jury could find for the plaintiff.  Every case presents a different constellation of facts and much of what courts must do is to weigh the facts.  Although the Supreme Court has provided a framework for this analysis (McDonnell-Douglas), in fact what courts do here is quite case-specific (and, as I’ve already suggested, quite unpredictable).  Judges in these cases don’t report every fact or piece of evidence.  They can’t.  And presumably, the decision to omit a fact is based on a determination that the fact, or piece of evidence, is not material.  So if a researcher were to go through the briefs and identify the facts mentioned by the lawyers but not the court, the researcher would still have to make a judgment whether the facts should have been mentioned.

So how to get around this problem?  One possibility is to focus on dissents.  In some cases, the dissents blow the whistle.  In Malacara v. City of Madison, for example, Judge Williams in dissent lays out the plaintiff’s substantial evidence to support his claim of discrimination, evidence that the majority simply ignores.  The problem is that such whistleblowing will point to only a fraction of the cases in which this problem arises, either because there is no dissent or because a draft dissent prompted discussion of the previously omitted facts.  This latter situation may be interesting for other reasons – does the presence of certain judges and the increased threat of whistleblowing affect the way or which facts are described?  This is a panel effect that I do not believe has been studied, but would be very difficult to observe, at least in the absence of judges’ papers.

There’s an additional facet to this problem that has to do with how the law develops, and that might provide some insight.  My colleague Mark Rosen has coined that expression “rulifying” to describe the way that the Supreme Court reliance on facts can turn those facts into elements of a legal standard or rule.  Putting these ideas together suggests an interesting possible empirical approach: is there less “whistleblowing” in areas of law that have relatively well-developed sets of facts courts are required to address versus areas of law that require judges simply to rely on their overall assessment of the evidence or facts – the totality of the circumstances?

The New Market for Federal Judicial Law Clerks

Avery, Jolls, Posner and Roth have written The New Market for Federal Judicial Law Clerks.  The Abstract:

"In the past, judges have often hired applicants for judicial clerkships as early as the beginning of the second year of law school, for positions commencing approximately two years down the road. In the new hiring regime for federal judicial law clerks, by contrast, judges are exhorted to follow a set of start dates for considering and hiring applicants during the fall of the third year of law school. Using the same general methodology as we employed in a study of the market for federal judicial law clerks conducted in 1998-2000, we have broadly surveyed both federal appellate judges and law students about their experiences of the new market for law clerks. This Article analyzes our findings within the prevailing economic framework for studying markets with tendencies toward early hiring - a framework we both draw upon and modify in the course of our analysis. Our data make clear that the movement of the clerkship market back to the third year of law school is highly valued by judges, but we also find that a strong majority of the judges responding to our surveys has concluded that non-adherence to the specified start dates is very substantial - a conclusion we are able to corroborate with specific quantitative data from both judge and student surveys. The consistent experience of a wide range of other markets suggests that such non-adherence in the law clerk market will lead to either a reversion to very early hiring or the use of a centralized matching system such as that used for medical residencies. We suggest, however, potential avenues by which the clerkship market could stabilize at something like its present pattern of mixed adherence and non-adherence, thereby avoiding the complete abandonment of the current system."

Similarly, Richard Epstein has an article in the Autumn 2006 Green Bag entitled "Ending the Mad Scramble: An Experimental Matching Plan for Federal Clerkships."