Empirical Legal Studies

Calls For Open Data Policies in Patent Law

Building off of Johnson & Johnson's recent decision to make all of its clinical data available to scientists around the world (favorable New York Time op-ed here), Dave Schwartz (Chicago-Kent) and co-authors make the case (here) that this same impulse should extend to empirical legal studies as well, particularly studies of patent assertion entities ("PAE"). An excerpt follows.

"Why is more information about PAE litigation not public? After all, the underlying data relates to litigation in the federal courts, and thus does not implicate privacy concerns like in the medical context. However, most of the raw data has been gathered and coded by private companies. For-profit businesses legitimately desire to use the information within their business and to prevent competitors and others from using commercially valuable information. That said, we believe corporate owners should release as much of the raw data (not merely descriptive statistics) as they can. To the extent that the raw data is not released or shared, society should be extremely cautious before relying upon it to make important public policy decisions."

ELS Outside of the Ivory Tower

ELS--at least in its applied form--increasingly attracts interest outside of the academy. [Now former] NYC Mayor Bloomberg developed and recently released a web-based application that allows users to explore recidivism rates for specific sub-groups of New York City criminal defendants and help predict defendants’ likelihood of re-arrest.

Data Analytic Recidivism Tool ("DART") "enables users, including criminal justice professionals, program planners and researchers to select a group of defendants based on factors like age, prior criminal history and details about their criminal cases. DART will then produce a graph displaying three different one-year re-arrest rates for the selected group, including the percentage re-arrested for any crime within a year; the percentage re-arrested for a felony within a year; the percentage re-arrested for a violent felony within a year; and a comparison to the citywide average."

PACER Data and Copyright Litigation

In a very brief (7 pp.) technical--though informative--paper circulating on SSRN, Empirical Studies of Copyright Litigation: Nature of Suit Coding, Matthew Sag (Loyola-Chicago) assesses variable coding reliability in a commonly-used database for legal scholars (PACER). Specifically, the paper focuses on "Nature of Suit" variable in the PACER records for empirical studies of copyright litigation. While Sag finds that the variable does not, in fact, capture all copyright cases, it nonetheless remains sufficient "for most purposes." Sag notes that the variable is especially suspect for copyright cases that involved pro se litigants and where copyright was not the primary litigated issue. Sag estimates that the "820 code" captures "80 to 85% of true copyright cases leading to written opinions."

SCOTUS & Judicial Decisionmaking Data Sets

Paul Collins, Jr. (Univ. N. Texas--Poli Sci) notes the public availability of the U.S. Supreme Court Confirmation Hearings Database that he and co-author Lori Ringhand used in their recent book, Supreme Court Confirmation Hearings and Constitutional Change (Cambridge Press, 2013).

"This database provides a wealth of information regarding the confirmation hearings of U.S. Supreme Court nominees held before the Senate Judiciary Committee. Based on confirmation hearing dialogue, the dataset includes information on the political environment surrounding the nomination, the issue and subissue areas being discussed, and the manner in which the nominees answer senators' questions. In addition, the database contains information on the discussion of judicial decisions at the hearings, including the name of the decisions and the courts that rendered the cases debated at the hearings."

Those who visit the site (here) will note that it includes a growing compilation of data sets--most germane to federal court judicial decisionmaking--linked to selected publications that exploit the data. Also, relevant Stata do files accompany many of the data sets. Not only does this site help disseminate useful data, but it also facilitates replication efforts.

Two final notes. First, when JELS editors (myself included) or JELS referees request data and do files from authors incident to manuscript reviews, Paul's web site includes examples of "best practices" that should be widely emulated.  Second, graduate and law students seeking to learn empirical methods will find these "ready-to-use" data sets invaluable.

Empirical Legal Scholars and "Special" Responsibilities

A recent post by David Schwartz (Chicago-Kent)--wondering whether empirical legal scholars should shoulder "special ethical responsibilities"--ignited a fascinating (and timely) discussion over at Concurring Opinions. Two reasons prompt Schwartz's concerns. "First, nearly all law reviews lack formal peer review. The lack of peer review potentially permits dubious data to be reported without differentiation alongside quality data. Second, empirical legal scholarship has the potential to be extremely influential on policy debates because it provides 'data' to substantiate or refute claims. Unfortunately, many consumers of empirical legal scholarship — including other legal scholars, practitioners, judges, the media, and policy makers — are not sophisticated in empirical methods."

Schwartz's concern focuses on what he calls "weak data." By that he means "reporting [results from] data that encourages weak or flawed inferences, that is not statistically significant, or that is of extremely limited value and thus may be misused." Specifically, "[t]he precise question I have been considering is under what circumstances one should report weak data, even with an appropriate explanation of the methodology used and its potential limitations."

Whether you agree with Schwartz or not, he raises an important question that warrants attention.

(Re-)Defined Data

Over at PrawfsBlawg, John Pfaff (Fordham) provides a cautionary reminder that most empiricists cannot hear often enough. Re-defining key variables can influence results and re-defined variables are frequently difficult to detect, particularly in large, complex, longitudinal datasets. That is, too often, secondary analyses are undertaken without necessary due-diligence involving the underlying data. Among Pfaff's take-aways:

"If nothing else, this is a strong warning against casually running empirical models, a growing problem in legal scholarship. Legal academics shouldn’t just get their IT departments to install Stata on their computers, download some data, and then start running some regressions. It  can take years to fully understand what a dataset looks like, what it is really measuring, its strengths and weaknesses. People who just run some quick regressions and then send them off to a law review are likely moving knowledge backwards, not forwards, since the risk of bad results is too great."

Supreme Court Database--Updated

An exceptionally helpful source of data for those interested in US Supreme Court decisions was recently updated to include data from OT2011. The Supreme Court Database (2012 release, v.01, here) "contains over two hundred pieces of information about each case decided by the Court between the 19[46] and 20[11] terms. Examples include the identity of the court whose decision the Supreme Court reviewed, the parties to the suit, the legal provisions considered in the case, and the votes of the Justices." An online codebook for this leading compilation of Supreme Court decisions (particularly for political scientists) can be found here.

Nitty-Gritty Practical Tips: Importing Excel Spreadsheets Into Stata (12)

Many--but not all--data sets analyzed in Stata began as Excel spreadsheets. To get to Stata's analytics requires importing Excel files. All too often, what should be a straight-forward task becomes anything but. A recent post on the Stata Blog walks readers through (with helpful screen-shot pictures) the data import process.

The "import excel" command in Stata 12 can help import Excel files into Stata by addressing two specific issues. First, Excel files often contain header and footer information in the first few and last few rows of a sheet, and you may not want that information loaded. A second common problem is that many column labels used in Excel spreadsheet files are invalid as Stata variable names (and therefore cannot be loaded into Stata).

Helpful "How-To" Guide for Using Adobe Illustrator to Enhance Graphics

As I am partial to elegance and subtly when it comes to graphs, tables, etc., the overly-complicated graphics command structures in many of the leading software packages can quickly frustrate even experienced users. Moreover, transforming graphical output into useful forms, such as PowerPoint slides, generates its own set of challenges.

Thus, I was delighted to find over at the Social Science Statistics Blog a link to a wonderfully helpful slideshow by Matthew Sigal (York), replete with practical tips on how to post-process graphs with Adobe Illustrator. Though the examples make specific reference to graphs produced in R, the advice is easily translated into other statistical software packages. Finally, while York's main point is to assist with the concrete task of enlisting Adobe into the service of better graphics, one implicit and broader point involves the importance of attending to the visual display of data.

Defunding Political Science at NSF

Late last night, on a nearly party-line 218-208 vote, the U.S. House passed an amendment (by Rep. Flake, R-AZ) to HR 5326 to "prohibit the use of funds to be used to carry out the functions of the Political Science Program in the Division of Social and Economic Sciences of the Directorate for Social, Behavioral, and Economic Sciences of the National Science Foundation."  The Monkey Cage has some of the relevant links.  Efforts like this have been mounted before -- most recently in 2009, by Sen. Tom Coburn -- but none have gotten this far.

The actual debate on the defunding amendment (all five minutes of it!) is here, in the CR.  I am not enough of a student of the appropriations process (or of legislative politics in general) to speculate on what might happen next.  But I do think that if I were Subra Suresh, Myron Gutmann, or the SBE Advisory Committee -- or, for that matter, the directors of the NIJ, any of the NIH agencies, etc. -- I would be very concerned about the precedent that this would set.  For Congress to begin micromanaging the NSF at the program level raises some serious concerns about the politicization of science.

The ACLU's Effectiveness as Amicus Curiae

One of my honors advisees this term has just completed her thesis, an empirical examination of the ACLU's ability to influence the U.S. Supreme Court through filing amicus curiae briefs during the Burger and Rehnquist Courts.  It's a careful study, combining qualitative and quantitative methods. The thesis has a number of interesting findings; I won't go into all the details, but I wanted to share this:

It's a plot of the odds ratio for the presence of an ACLU brief (from a series of justice-specific logistic regressions that also included a host of controls) where the outcome variable is a pro-ACLU vote.  The vertical line at 1.0 corresponds to "no marginal association;" values less than one are negative associations, while values above one are positive. (I'm omitting the CIs because they make the plot very messy.)

The interesting thing, of course, is that justices widely perceived as more politically liberal also tend to be those that are most "influenced" by the ACLU's amicus briefs.

Also, to all those NYC firms out there: My advisee, Shannon Azzaro, will be starting law school at Fordham in the fall, with a focus on intellectual property.  I understand that her 2013 summer is still free...

An Updated Database of State Tort Law Reforms (4th)

Ronen Avraham (Texas) recently alerted me to an update of one of the more helpful resources around for those who study tort reform. Specifically, Database of State Tort Law Reforms (DSTLR 4th) seeks to create "one 'canonized' dataset will increase our understanding of tort reform’s impacts on our lives." A fuller (though excerpted) description follows.

"This manuscript of the DSTLR (4th) updates the DSTLR (3rd) and contains the most detailed, complete and comprehensive legal dataset of the most prevalent tort reforms in the United States between 1980 and 2010.... The dataset records state laws in all fifty states and the District of Columbia over the last several decades. For each reform we record the effective date, a short description of the reform, whether or not the jury is allowed to know about the reform, whether the reform was upheld or struck down by the states’ courts, as well as whether it was amended by the state legislator."

More Practical Data Set Merging Tips (From Stata)

Following up on an initial post (previously described here), the Stata blog has a second post on data set merging tips. The second post focuses on multiple key data set merges. Multiple-key merges are required when more than one variable is needed to uniquely identify observations in your data. This situation frequently arises in panel or longitudinal data sets.

Helpful, Practical Stata Tips

While I take no side (or sides) in the proverbial "Stata v. SPSS v. R" wars (I use all three), I want to note Stata's recent venture into the Blog universe. Though it's far too early to see if the Stata blog will succeed, it's an interesting development nonetheless. A recent post provides a helpful step-by-step guide to one of the more common (yet error-prone) data tasks: merging data sets. The explanation is helpful and includes all too common pitfalls.

Data on the "Employed at Graduation" U.S. News Rankings Input

By Bill Henderson & Andrew Morriss

Yesterday, Bob Morse of U.S. News published a blog post in which he signaled a change in the law school rankings methodology, specifically with regard to employment.  The prevailing view on the law school administrator list-serves (which nearly a dozen people have forwarded to us) is that U.S. News will be increasing the weighting of "employed at graduation," presumably because U.S. News Editor Brian Kelly sent a letter to law school deans--reprinted in Bob's blog post--discussing the importance of employed-at-graduation as a metric.

We have zero inside information, but we are willing to bet a substantial sum that any methodology change will be in a completely different direction.  Here is why.  Over the last decade, fewer and fewer schools have been supplying U.S. News with employed-at-graduation data.  Employment at graduation is not a statistic required or collected by the ABA; as such, its accuracy cannot be checked through cross-reference to the annual ABA-LSAC Official Guide. 

But much more significantly, when a school fails to provide this data, U.S. News has--up until now--imputed the figure based on employment at 9 months.  (Kudos to Ted Seto for unraveling this mystery.  See Understanding the U.S. News Law School Rankings.)  Crudely speaking, the magazine applied a roughly 30% discount rate on the employed at 9 months figure.   Earlier this year, Paul Caron suggested that if a school's employed-at-graduation rate is more than 30% lower than its employed at 9 month rate, it is "rankings malpractice" to supply U.S. News with the data.

As readers can see from the above chart (generated by Paul Caron in his rankings malpractice post), a large proportion of law schools have figured out the payoffs.  Over the last decade, the percentage of non-reporting schools has skyrocketed. With this information in mind, Bob Morse's blog post may seem less cryptic:

In an effort to make our law school employment data more reflective of the current state of legal employment, U.S. News has modified how we calculate the employment rates that are used in the new law school rankings.  ...

U.S. News agrees with the efforts of Law School Transparency to improve employment information from law schools and make the data more widely available.

If the goal is (a) to utilize data that better reflect reality, and (b) provide greater transparency and access to such data, it makes no sense to increase the weight of an input (employed-at-graduation) that is either withheld by law schools or is heavily gamed.  This latter point is made by U.S. News Editor Brian Kelly in his letter to the deans:

[E]mployment after graduation is relevant data that prospective students and other consumers should be entitled to.  Many graduate business schools are meticulous about collecting such data, even having it audited. The entire law school sector is perceived to be less than candid because it does not pursue a similar, disciplined approach to data collection and reporting.

At U.S. News, we work to make meaningful and fair comparisons, based on industry-accepted data. ...

To eliminate some of the gaming that seems to be taking place, we have changed the way we compute employment rates for the rankings due out March 15.  In addition, we will also be publishing more career data than we have in the past in an effort to help students more completely understand the current state of legal employment. We think more still needs to be done.

So Kelly is saying that employed at graduation data are important, and the magazine is tired of being gamed.  Therefore, we think two methodogical changes have a good chance of being implemented:

• U.S. News is likely to heavily penalized schools that withhold the employed at graduation data.  Going forward,  the imputation may be far more negative than -30% off of employed at 9 months.  A drop in rankings will stop in its tracks the non-response problem.

• Regarding perceptions of gaming, it is possible that U.S. News has formulated a way to quantify how many jobs at graduation map onto full-time professional jobs that require a law degree.  For example, the ABA Official Guide provides lots of comparable data by practice setting.  Law firms, judicial clerks, and government jobs could be weighted more heavily than business or academic jobs.  Unknown may also be treated as 100% unemployed rather than the current 25% presumption of employment.  Such changes would have the law schools scrambling to report better numbers in higher weighted categories rather than just finding ways to goose up the employed-at-graduation and employed-at-9 months figures.  Remember that Bob Morse explicitly endorsed the Law School Transparency movement.

We would like to suggest to our colleagues in the legal academy that we are approaching an endgame.  Here is the reality:  prospective students are not being given an accurate picture of their future employment prospects.  Why? Because we are all focused on filling next year's class with as many high credential students as possible, thereby protecting our school's place in the pecking order.  Our focus is so shockingly narrow that, from the outside looking in, it appears that our intent is to deceive incoming students.  Brian Kelly's letter to the deans essentially makes that point--law schools fall short on candor and ethical behavior.

The numbers that get submitted to U.S. News include many graduates who are technically employed but often significantly underemployed, often at positions that don't require law degrees.  Finer grained data get reported to NALP, but they are never published on a school-by-school basis.  If these data were released, prospective students may not fully process the information--that is an argument that we often hear law professors make.  But that does not alter our duty to provide "basic consumer information ...  published in a fair and accurate manner reflective of actual practice."  ABA Accreditation Standard 509.

At some point, all our lawyerly rationalizations will come to a bad end because a governmental agency or a court is going to challenge our right to self-regulation, thus ushering in a truly disgraceful chapter in the history of American legal education.  

Now is one of the very few moments in our careers as academics where we have to make hard choices and demonstrate that we warrant the trust and respect of our tenured positions.   Through our governance organizations (ABA, LSAC, NALP, AALS), we need to implement a system of complete transparency on employment outcomes.  If the system has real teeth, it will force us all to work very hard to ensure we are delivering value commensurate with the tuition dollars we collect.

It's the end of the road.  We likely have one last chance to get it right.

JEDI

Posting Andrew's call regarding his course on empirical legal scholarship made me wonder whether our readers are aware of his Judicial Elections Data Initiative?  If not, take a look.  This is GOOD stuff.

http://jedi.wustl.edu/

Data & "Self-Quantification"

Over at Concurring Opinions, Scott Peppet notes an explosion of personal data gathering and considers the obvious privacy implications flowing from "self-quantification."

Law Schools, Students, and "Legal Employment": Supply and Demand

Over at Balkinization Brian Tamanaha (Wash U) posted graphs endeavoring to describe law student enrollment and BLS data on legal employment trends (2001-09). Visually, the two graphs support Brian's assessment: "Law schools thus responded to the worst recession in the legal market in at least two decades by letting in more law students." As Brian Leiter (Chicago) notes, however, "One can't tell, though, from the second chart [and the underlying BLS data] what portion of the downturn in 'legal employment' is a reduction in the employment of attorneys as opposed to other law-related employees."

The Empirics of Family Law: Influence of Child Support Guidelines on Divorce

In Child Support Guidelines and Divorce Rates, Margaret Brinig (Notre Dame) and Douglas Allen (Simon Fraser) draw on a large dataset (National Longitudinal Survey of Youth (NLSY)) to assess how variations in child support guidelines influence decisions to divorce. The paper's abstract follows.

"A child support guideline is a formula used to calculate support payments based on a few family characteristics. Guidelines began replacing court awarded support payments in the late 1970s and early 1980s, and were later mandated by the federal government in 1988. Two fundamentally different types of guidelines are used: percentage of obligor income, and income shares models. This paper explores the incentives to divorce under the two schemes, and uses the NLSY data set to test the key predictions. We find that percentage of obligor income models are destabilizing for families with high incomes. This may explain why several states have converted from obligor to income share models, and it provides a subtle lesson to the no-fault divorce debate."

Hot off the Data Presses

ICPSR has just made available Wave 1 of the After the JD database.  From the study description:

The After the JD project is designed to be a longitudinal study, seeking to follow a sample of approximately 10 percent of all the individuals who became lawyers in the year of 2000. It is the largest and most ambitious study ever undertaken by researchers of legal careers aiming to track the professional lives of more than 5,000 lawyers during their first 10 years after law school.

Updated State Tort Law Reforms Dataset (v.3)

Ronen Avraham (Texas) has updated a tremendously useful resource for those assessing the impacts of various tort reforms. In the Database of State Tort Law Reforms (3rd), Ronen sets out to establish "one 'canonized' dataset will increase our understanding of tort reform’s impacts on our lives." Direct access to the dataset (in Excel) is found here. A more complete description of the project follows:

"... DSTLR (3rd) updates the DSTLR (2nd) and contains the most detailed, complete and comprehensive legal dataset of the most prevalent tort reforms in the United States between 1980 and 2008. ... The dataset records state laws in all fifty states and the District of Columbia over the last several decades. For each reform we record the effective date, a short description of the reform, whether or not the jury is allowed to know about the reform, whether the reform was upheld or struck down by the states’ courts, as well as whether it was amended by the state legislator."

Data Sharing Norms & Variation Across Fields

Jeff Yates (SUNY Binghamton--Poli Sci) ignited an important discussion (here) about data sharing norms and possible variations across fields. Jeff notes:

"In political science there are very strong professional mores to share data with other researchers. In fact, it is usually expected immediately after publication of your first article using the data if not before that time (e.g. after presenting a working paper at a conference). I profess some ignorance of the social mores on data sharing in empirical legal studies, but from my few conversations on this point, I think that they might be somewhat different."

Setting aside scholarly norms (or, better yet, aspirations), Jeff also wonders whether IP doctrine tugs norms for empirical legal scholars.

Greater Access to More Municipal Data Sets

Maya Sen, over at the Social Science Statistics Blog, posts about (with links) a wonderful emerging trend involving increased access to municipal-level data.

Data Preservation

Setting aside the politics surrounding the global warming debate, Eugene Volokh's discussion (here) about discarded raw data takes up an issue important to every empiricist--data preservation.

Supreme Court Database Website

Good news -- the new NSF-funded Supreme Court Database Website is now up and running, and I can't imagine this won't make the database available to thousands more users via its easy-to-use interface.  Not only can you download the most current version of the database and its companions, but you can also perform analyses right on the website.  This is good stuff; check it out! 

http://supremecourtdatabase.org 

Bias in Spaeth Data?

With a hat tip to the Monkey Cage, I thought folks might be interested in taking a look at this paper.  I know my fellow editor Carolyn has done some research on the Spaeth data as well -- perhaps she'll weigh in on this research?  (FYI, I have not yet read the Harvey/Woodruff paper.)

Updated Judicial Common Space Scores

here. Courtesy of Profs. Lee Epstein, Andrew Martin, Jeff Segal, and Chad Westerland. JCS attempt to provide preference estimates for Supreme Court justices that are directly comparable to preference measures of Courts of Appeals judges, members of Congress, and the President. Our data extend through 2008 and correspond with the most recent version of Keith Poole's Common Space scores.

Hello!

Thanks to the Empirical Legal Studies crew for letting me guest blog here. Consistent with the motto of the blog, I hope to use my time with ELS to bring some method to the madness that seems likely to swirl around Sonia Sotomayor’s confirmation hearings this summer.

 

For the past two years I have been compiling a dataset capturing all of the statements made by senators and Supreme Court nominees at the confirmation hearings held by the Senate Judiciary Committee.  The dataset starts with the Felix Frankfurter’s hearing and ends with Justice Alito’s. I have coded by issue area all statements made by the nominees and senators. Each unit of analysis includes information about the political party of the questioning senator, the appointing president, and the committee chair. The dataset also captures instances where a court case was discussed by name, and includes variables indicating whether the statement being coded addressed constitutional interpretation, statutory interpretation, or federalism.

 

I hope to use this data to provide real-time commentary on the confirmation hearings this summer.  Because Supreme Court vacancies occur so infrequently, our discussion of the confirmation process often lacks historical perspective. I hope this data can help rectify that, by bringing concrete information to conversations about what has – and has not – been “the norm” in this process. 

 

I also hope that the ELS community will provide feedback about how to use and improve this dataset. This is in many ways a ‘test run’ of the data, so I look forward to hearing your comments about it. I’m also open to suggestions regarding what type of information you think would be interesting to pull out of the dataset.

 

First up: What makes a confirmation contentious?

The Dataverse

I just finished posting some data and documentation on the Dataverse Network.  This is a wonderful -- and remarkably easy to use -- resource that provides scholars with a virtual archive for their data.  It allows you to make your data publicly available in a permanent format with a stable URL -- useful if, for example, you change institutions.  You can even incorporate links to your data into your own website, branding the material as your own, without having to worry about hosting and maintaining it.   If other scholars use or rely on your data, there is a formal citation that they can use, giving you appropriate scholarly credit.

And of course the Dataverse is a great resource for scholars who might want to replicate other people's work or rely on their data.   Thanks to Harvard's Institute for Quantitative Social Science for creating and making available this resource.  You can read more about it here, in a post by Gary King at the Social Science Statistics Blog.

Judicial Pay Debate Data

Over at the Volokh Conspiracy Eric Posner brings logitudinal data to the judicial pay debate. The data come from a paper on the same topic, Are Judges Overpaid?: A Skeptical Response to the Judicial Salary Debate, by Stephen Choi (NYU), Mitu Gulati (Duke), and Eric Posner (Chicago).

Effect of "High Score" Rule on Volume of LSAT Repeaters

A couple of years ago, Paul Caron flagged an obscure ABA rule change that required law schools to report the highest LSAT score of an admitted student versus the prior practice of averaging.  Paul and Moneylaw blogger Tom Bell foresaw a likely surge in the number of repeat test-takers--but without any appreciable benefit.  Repeat score are, in fact, less accurate in predicting law school grades -- though, let's not kid ourselves, law schools are not looking at the LSAT anymore for predicting 1L performance.  It also costs a lot of time and money.  And if upper class white kids are better able to afford test preparation courses, it is likely to exacerbate the racial performance gap. 

Well, the data is in for the October 2008 cycle.  It is indisputable that applicants are figuring out the implications of a low LSAT score in the US rankings era (fewer admissions letters, few scholarship dollars) and the potential upside/no downside of taking the exam again. Despite a -1.7% drop in first-time takers, the repeater volume is up 16.8%.  In the Northeast, where the positional competition is the most intense, there has been a staggering 33.7% increase. 

From an individual perspective, I know it makes sense to take the test a second or third time. Indeed, it is comforting to many to have that option.  But in the aggregate, this policy really just opens the door for a protracted zero sum game.  If law schools were making better decisions because of the second or third scores, the additional time and expense could be justified.  The second scores, however, are less reliable than the first and second combined. 

Understanding these dynamics, the regulator (the ABA Section on Legal Education and Admission to the Bar) is supposed to set rules that are in the best interests of the students and the profession -- not law schools or testing agencies. Sam Stonefield of Western New England Law wrote a very detailed objection to this policy. He was prescient.  I appreciate the time he took to spell it all out.

More From BJS: State Civil Litigation Trends, 1992--2005

Supplementing one of the leading sources of data for state civil litigation activity in the United States, the Bureau of Justice Statistics (BJS) recently released a report, Civil Bench and Jury Trials in State Courts, 2005, which provides an important snapshot and illustrates litigation trends since 1992. The report discusses general civil cases (tort, contract, and real property) concluded by a bench or jury trial in a national sample of jurisdictions in 2005. Topics include the types of civil cases that proceed to trial, the differences between civil cases adjudicated by judges or juries, and the types of plaintiffs and defendants represented in civil trials. Also, the report covers plaintiffs.  Key findings include:

• In 2005 plaintiffs won in more than half (56%) of all general civil trials concluded in state courts. The plaintiff was significantly more likely to win in a bench trial compared to a jury trial. Among all plaintiff winners the median final award was $28,000. Approximately 4% of all plaintiff winners won $1,000,000 or more. Contract cases in general had higher median awards ($35,000) than tort cases ($24,000).

• The total number of civil trials declined by over 50% from 1992 to 2005 in the nation’s 75 most populous counties. Tort cases decreased the least (40%) while real property (77%) and contract (63%) cases registered the largest declines.

• In the nation's 75 most populous counties, some tort case categories have seen marked increases in their median jury awards. This was particularly the case for product liability trials, where the median awards were about 5 times higher in 2005 than in 1992 and for medical malpractice trials, where the median jury awards more than doubled from $280,000 in 1992 to $682,000 in 2005.

The Drift Toward Pure Numbers Admissions

Law schools are part of a production function for entry level lawyers.  Therefore, if law schools alter their admissions practices, the character and complexion of the hiring pool can shift in significant ways.   

On the input side, the data are crystal clear:   over the last fifteen years, the rankings arms race has pushed U.S. law schools toward a pure numbers approach to admissions.   The more interesting question, however, is whether prestige-conscious law firms are now, inadvertently, experiencing any fallout.   First the data.

Law schools operate in an environment of supply and demand and are famously counter-cyclical.  When Silicon Valley was booming in the late 90s, law school applicants plummeted.  When the economy faltered in the early 90s or after 9/11, applicants spiked.  Therefore, to examine how admissions practices have changed over time, it is important to pay attention to the underlying applicant pool.  Below are trend lines for median LSAT scores by USNWR rank for 1994 and 2007, which reflect classes that entered in the fall of 1993 and 2006 respectively.  During those two admissions cycles, the number of applicants was virtually identical:  89,600 (1994) and 88,700 (2007).

Obviously the blue line (2007) is higher than the orange line (1994).  In fact, despite slightly fewer law school applicants, the average median LSAT increased by 2.18 points (std. dev. of 1.99). For the record, only three schools fell out of Tier 1 between 1994 and 2007.  And it cannot be explained by the ABA policy shift that instructs law schools to no longer average LSATs for reporting 25th, 50th, and 75th percentile figures, thus slightly pumping up the volume of high LSAT scores.  That change was not enacted until the summer of 2006. 

Here is the same analysis for UGPA (1994 data came from the Princeton Review, 2007 from the ABA):

Although we might chock some of the higher UPGAs (avg. of +.17, std. dev. of +.12) on grade inflation between 1994 and 2007, it is likely that schools were also trying to maximize this number. More after the jump ... .

The Limitations of the Spaeth Supreme Court Database

I have posted on SSRN my article, forthcoming in Hastings Law Journal, Coding Complexity: Bringing Law to the Empirical Analysis of the Supreme Court.  This article examines the well-known and widely-used U.S. Supreme Court Database (created by Harold Spaeth) – and most recently mentioned here –  and addresses the Database’s limitations particularly for those interested in law and legal doctrine.
The key point of the Article is that the Database does not contain complete or accurate information about law and legal doctrine as they appear in Supreme Court opinions.  Given Harold Spaeth’s own purposes in creating the Database, these limitations may not be surprising -- although they do raise at least some challenges to his attitudinal model.  Unfortunately, however, they are frequently misunderstood.  Scholars all too frequently use the Database in ways that it simply cannot support, leading to the possibility of invalid or unreliable results.  This post summarizes the Article’s main arguments. 
The primary challenges presented by the Database involve the coding for the “issue,” “issue area,” and “legal provision” variables.  As the names of these variables suggest, they are frequently used by researchers interested in studying law and legal doctrine.  Yet, the coding protocols for these variables (as set forth in the Codebook are not conducive to such research.  Some of the limitations of these variables include:
(A) The “issue” variable is not, despite its name, designed to identify any legal issues in a case.  Rather, it is designed to identify the “public policy context” of a case.  A case like Schenck v. Pro-Choice Network of Western N.Y. is one example.  In Schenck, a group of abortion protesters challenged an injunction limiting their activities as violating the First Amendment.  The only legal issue in the case involves the First Amendment and the limits it places on judicial power.  But the Database codes the case as having an issue of “abortion” because that is the factual, or “public policy” context in which the case arises.
(B) The coding contains a strong presumption of assigning each case only a single issue.  So the Database does not add a First Amendment issue code to the coding of Schenck.
(C) The issue codes are quite underinclusive and somewhat dated.  For example, there are no codes for immunities, for sexual harassment, or for the dormant commerce clause.
(D) Each of the approximately 260 issue codes is classified into one, and only one, of 13 “issue areas.”  In some cases, the classification makes no sense.  For example, in Markman v. Westview Instruments, Inc., the Court addressed the question of whether patent claims construction is a question for the judge or the jury; that is, whether there is a 7th Amendment jury right.  The Database classifies Markman as a case about the right to a jury trial, but that code, which does not distinguish between civil and criminal jury rights, is located in the Criminal Procedure issue area.
(E) The legal provision code does not identify cases or judge-made legal doctrines.  It is limited to identifying statutes, constitutional provisions, and court rules.
(F) The coding protocols provide that only legal provisions mentioned in a case’s syllabus should be identified.  But the syllabus – a short summary of the case – is akin to headnotes.  It is not officially part of the case, it is not written by the justices or their law clerks, and it cannot be cited by lawyers or judges.
To some extent, misuses of the Database are likely due to differences in the ways that different disciplines (political science and law) use the same words.  To some extent, misuses stem from scholars failing to evaluate their research design in light of the Database’s coding protocols, which are described in the Database’s Codebook.  In my Article, I provide a series of examples of research project that fail to adequately take account of the Database’s limitations and that therefore produce results that may be inaccurate.
To further explore the limitations of the Database and to experiment with more legally nuanced types of coding, I undertook a Recoding Project of a random sample of 10% of the cases from the last Rehnquist natural court.  The details of the coding project are, of course, explained in the Article.  Among other things, I redefined “issue” to mean legal issue, I expanded and rearranged the lists of issues and issue areas, I put no limit on the number of issues that could be coded per case, I redefined legal provision to include seminal cases and legal doctrines, and I identified legal provisions by looking at the opinions themselves, not just the syllabi.
Some of the key findings of the Recoding Project include:
(1) I identified an average of 3.7 issues and 2.4 issue areas per case, rather than the single issue and issue area per case identified in the Database.
(2) I identified an average of 2.2 as many legal provisions per case as did the original Database.
(3) A surprising number of legal provisions that I identified should have been identified in the Database because they were mentioned in the syllabi.
(4) In both issue and legal provision coding, the “missing” codes – those that I identified but that the Database did not – disproportionately related to structural and jurisprudential issues, including procedure, the powers and operations of the federal and state governments, and the relationship between different branches of government.
These and other findings have a variety of implications for researchers working with the Database.  Chief among these is the importance of not drawing conclusions about the Supreme Court’s cases by looking at the numbers and types of issues, issue areas, and legal provisions coded.  Researchers all too often rely on such information to draw conclusions about case complexity or about the number of issue “dimensions” in the cases.  In other words, researchers sometimes point to the Database to justify their assumptions that most Supreme Court cases involve only a single issue.  But as the Article demonstrates, this single-issue coding is -- or at least may well be -- an artifact of a coding protocol that presumes that each case should be assigned only a single issue, so such conclusions are circular.  A second important implication is that the Database’s issues and issue areas do not accurately identify all cases involving particular legal issues and that not all cases with a particular issue or issue area code in fact involve the legal issues that a researcher might presume from the names of those codes.

Spaeth's S.Ct. Coding (07 Term)

Harold Spaeth has completed his coding for the Supreme Court's 07 Term. The dataset ("allcourt") is available for public download at the University of South Carolina Judicial Research Initiative (JuRI). To access the dataset please visit www.cas.sc.edu/poli/juri.

If Yale is #1 in U.S. News, Is it the Best Law School?

In a provocative post entitled "Is the End Near for Yale's Dominance", Brian Leiter reports one insider's assessment that Yale may lose three or four additional faculty members.  If that happens, surely Harvard, with its recent lateral hiring sprees, will be the best law school in the country--right? Brian thinks that predictions of Yale's decline are premature.  I agee.  But in the process Brian implicitly highlights an interesting problem: what exactly does it mean to be the "best" law school? 

Here are the vexing facts:  on a per capital basis, Yale places more people in academia and Supreme Court clerkships than any other law school;  Yale's acceptance rate is 7.3% versus 11.8% for Harvard; yet, over the last decade, the average U.S. News academic reputation score for the two schools are exactly--yes, exactly--equal: 4.840 for Harvard, 4.840 for Yale. 

Is it possible that Yale is #1 because, well, Yale is #1 -- and has been every year since USN began publication?  Brian refers to U.S. News "small school bias".  He is right.  Because of Yale's massive endowment and small student size, it enjoys a per-pupil expenditure that is roughly 1/3 its total tuition price.  According to a simulation model of the 2008 U.S. News rankings, which Andy Morriss and I recently constructed, Harvard would not overtake Yale even if: 

• Harvard's median LSAT climbed to 180 and its median UGPA hit 4.0;
• Harvard's academic and lawyer-judge reputation scores were both a perfect 5.0;
• Harvard's acceptance rate plunged to less than 5%.

In fact, even with these changes, Yale would still have a nice leadership cushion.  [Note:  Similar anomalous math was noted by Ted Seto in his classic essay, Understanding the U.S. News Law School Rankings, SMU L Rev (2007).]

Yet, Yale's dominance keeps things simple.  Applicants signal their elite status by enrolling at Yale.  Judges, in turn, derive prestige by hiring Yale graduates, even though they mockingly complain that Yale clerks know very little law.  And faculty favor Yale graduates because it validates our own sense of eliteness and institutional upward movement.  We can rationalize Yale's dominance in terms of scholarship, but the real endgame is the allocation of positional goods.  It is so easy to get too caught up on the hamster wheel of envy and prestige without realizing that the energy expended does not necessarily produce anything of lasting social value.

Of course, each of us is free to determine our own merit criteria.  I think  the "best" law school is the one where faculty are willing to make inordinate personal sacrifices for the benefit of the collective enterprise--and where aspiring lawyers leave the law school skilled, confident, ethical, and ensconced in a powerful professional network that opens doors and values public service.  In turn, alumnus are sufficiently grateful for the transformative experience they received that they are willing to underwrite the law school's mission and subsidize this opportunity for future generations.  This vision requires a greater focus on internal rather than external metrics.  For us human beings, that is no easy trick.

Class of 2007: A More Extreme Bi-Modal Distribution

NALP just published its 2007 edition of Jobs & JD's.  One topic of interest to students, lawyers, law firms, and legal educators is the change in salary distribution from 2006 to 2007.  The now famous 2006 bi-modal distribution was vivid evidence that the U.S. legal profession is undergoing significant structural change.  As shown in the graph below (from this NALP webpage entitled "Another Picture Worth a 1,000 Words"), the underlying stressors are even more pronounced for the class of 2007. 

The sample is based on 23,337 law school graduates from the class of 2007 who reported salary information.  Note, however, that 197 ABA-Accredited law schools graduated 43,518 students in 2007.  Although we know the types of jobs taken by 40,416 grads, only 57.7% of this group provided salary information.  If I had to wager on the direction of underreporting, I would predict it was under-inclusive of graduates with lower salaries and those who did not pass the bar.  Why?  Aside from the human psychology that it is easier to share flattering rather than embarrassing information, the roughly 7,500 jobs under the second mode are fairly close to figures I have seen from ALM and NALP data, which are provided by large law firms rather than individual students.  See, e.g., charts in this NLJ article.

This bias, however, is not necessarily good news.  In the above graph, 32.5% of the law graduates took jobs with starting salaries in the $100K+ range; but the true percentage for the class of 2007 is probably lower.   Some facts and then one normative observation.  The facts first:

• 91.9% of 2007 graduates were employed 9 months after graduation, which compares favorably to 2006 (90.7%), 2005 (89.6%), 2004 (88.9%), and 2003 (89.0%).  I would like to believe these numbers are trustworthy. 
  
• 76.9% were in jobs that required bar passage.  [It would be useful to disaggregate the jobs in the remaining 23.1% of law school graduates.  Who are these students?  How many entered law school with no intention of practicing law? ]
• The median salary in the above distribution is $65,750; the mean is $86,396.  But these measures of central tendency are not reliable guides of future earning power.
• 38% of all starting full-time salaries were less than $55,000 per year, including 18% of all jobs in private practice, 27.5% in business, and 70.0% in government (excluding judicial clerkships).
• 79.6% of law firm jobs in NYC, 80.3% in Washington DC, and 74.9% in Boston were in firms with 100+ lawyers.  Even in Indianapolis, 50.4% were in 100+ lawyer firms.  Wow! those are big numbers.

See also NALP Press Release, July 24, 2008.  On the normative front, I have a simple thesis:  the bi-modal distribution is bad for students, bad for law firms, bad for clients, and bad for law schools.  [When I showed the 2007 distribution to one law school dean, she shielded her eyes!]:

• Students.  It is bad for students because at $160,000 per year, many corporate clients will ask that you not be assigned to their matters.  And if your initial work experience is document review, a $160K job can quickly become a dead-end because your skill set is not growing with your billing rate (avg. 1st yr billing rate in a $160+ firm is $225 to $255/hr).   So the atmosphere among associates at $160K+ firms is probably becoming more competitive.  It would be better in the long run to start at $95K, learn your craft, and become a great lawyer who commands top dollar.  And young lawyers should think long run.

• Clients.  This is bad for clients because the short term solution of requesting only midlevels and partners will eventually constrict the supply of incoming legal talent.  When clients and law firms try to externalize the cost of mentoring and training--here I mean observation, contact, and feedback from partners and clients--associates are more likely to leave.

• Law Firms.  Actually the bi-modal distribution is only bad for firms trying to keep pace with the Am Law 200 salary pay scale.  In contrast, boutiques and organizations like Axiom will find general counsel more interested in their value proposition.  For Am Law 200 firms, the difficulty is getting partners to commit themselves to the future of the firm by spending more time and money investing in associates.  This will reduce attrition and protect the brand.   But the $160K+ cost structure provides partners with strong incentives to bill hours rather than investing in the long term future of the firm.

• Law Schools.  The economics of the bi-modal distribution take the pressure off elite law schools--indeed, they can raise tuition! Thus, for many law professors, the best outcome is lateraling into a Top 15 law school.  But more/better law review articles--a precondition of a lateral offer--is not going to solve the difficult institutional problems of lower ranked schools.  Now more than ever, all law faculty members need to understand the structural shifts taking place in our profession.  When faculty at Harvard and Yale ignore these changes, it does not mean that these changes are not important.  It just means that Harvard, Yale, et al. are not affected.

I don't have any solutions to these issues, though I did write up some useful insights in my prior post, "Part II: How law firms misapply the 'Cravath System.'"  Our situation reflects difficult collective action and coordination/signaling problems.  For example, how a firm gracefully bows out of the salary wars is an immensely difficult problem.  I do think, however, that permitting nonlawyer investment would provide law firms with the financial wherewithal (and psychological courage) to experiment with more innovation.  And that would be good.  Larry Ribstein's scholarship is now more timely than ever.  See, e.g., here and here.

When I was an interim associate at Sidley & Austin the summer after the 2000 salary wars, a partner told us that "we are all going to hell" based of the jump in salaries from $95K to $125K.  I now worry that he may have been right.

Part II: How most law firms misapply the "Cravath system"

In my last post, I discussed the linkage between the bimodal distribution and the emphasis on credentials under the "Cravath system".  I also stated that most law firms misunderstood the internal logic of the original Cravath model and promised to elaborate in a subsequent post.   This is the promised entry. 

One note of context:  this post is not a history lesson.  The Cravath system reflects a profoundly powerful method of developing human assets.  Cravath started with very good associates/inputs  and turned them into truly exceptional lawyers who were in high demand by clients and other firms.   Moreover, the Cravath system required lawyers to work together collaboratively to further the clients' interests.  This resulted in efficient and highly effective legal services that engendered the abiding loyalty of clients and more demand for the firm's services. See Results or Résumés at 4 & n. 13 (discussing concept of firm-specific capital). In other words, under the true Cravath system, everyone comes out ahead. Two caveats:  (1) the first-mover--here, Cravath 108 years ago--garners the most benefit; (2) if a firm neglects a key element--e.g., investing in associates--the model generates no competitive advantage.

[Sources: The Cravath system described below come primarily from Robert Swaine's 1948 history of the firm and other contemporaneous sources from the 50s, 60s, and 70s, which I will cite as appropriate.]

Recruiting Elite Law School Graduates

One of the hallmarks of the Cravath system is the recruitment of elite law school graduates.  As of 1948, Cravath, Swaine & Moore and its predecessor firm had employed a total of 454 law school graduates as associates.   Of this total, 67.7% attended Harvard (128), Columbia (124), or Yale (54).  According to Swaine, "in recent years there has been an increasing number from the law schools of the Universities of Virginia and Michigan."  These two schools rounded out the top five :  (UVA 30, Michigan 26).

Cravath's emphasis on credentials, however, had a clear economic logic that was designed to compensate for the deficiencies of early 20th century legal education.  During this period, most law schools required little or no college education.   In contrast, Harvard, Columbia, and Yale grads typically had a college degree before entering law school.  Swaine writes,

Cravath believed that disciplined minds are more likely to be found among college graduates than among men lacking in formal education ... .

Cravath believed in seriousness of purpose--a man with a competent mind, adapting to practicing law according to Cravath standards, should have made a good scholastic record at college.  But he recognized, without full approval, the tradition of the early decades of this century--that "gentleman" went to college primarily to have a good time and make friends.  Hence, while a good college record was always a factor in favor of an applicant, lack of such a record was not necessarily an excluding factor. ... [I]n the stern realities of the depression of the '30s, however, college records of applicants came to have added importance.

Yet, "[f]or a poor law school record Cravath had no tolerance."  Candidates who "had not attained at least the equivalent of a Harvard Law School "B" either had a mind not adapted to the law or lacked purpose and ambition ... ."  Thus, the "first choice" was a "Phi Beta Kappa man from a good college who had become a law review editor at Harvard, Columbia or Yale."

Note, however, that Cravath's emphasis on credentials had a clear business purpose designed to compensate for the limitations of legal education.  During the first half of the 20th century, going to an Ivy League law school did not guaranteed legal aptitude. Prior to advent of the LSAT in 1948, college grades were the only predictors of success in law school.  In 1955, The "LSAT Handbook" included cross-tab tables of  LSAT scores versus law school performance for several individual law schools.  At Harvard Law, roughly 1/3 of the class scored below the 50th percentile.  On the west coast, UC Berkeley had a similar wide range of LSAT scores.  See The Law School Admission Test and Suggestions for Its Use (ETS 1955).  The clear relationship between LSAT and grades subsequently  encouraged law schools to revamp their admissions criterion.   But that process took decades.  See Lunneburg & Radford, The LSAT: A Survey of Actual Practice, 18 J. Legal Educ. 313 (1965).

In a talk at Harvard Law School, Cravath stated that a successful "lawyer of affairs" (aka corporate lawyer)  assumed "the fundamental qualities of good health, ordinary honesty, a sound education and normal intelligence."  On top of these attributes, a candidate must have "character, industry and intellectual thoroughness, qualities that do not make for charm but go far to make up that indefinable something that we call efficiency.  Brilliant intellectual powers are not essential."

More after the jump...

How the "Cravath System" Created the Bi-Modal Distribution

The bi-modal distribution (graphic to the right) continues to generate interest in the blogosphere.  See, e.g., Greg Mankiw, Right Coast, Broken Symmetry.   The  chart summarizes the starting salaries for lawyers who graduated from law school in 2006.  One reason the bi-modal structure is so jarring is that it demonstrates that measures of central tendency, such as average or median, are not necessarily reliable guides for law students' future earning power.  In conventional labor markets, that disconnect is rare.

NALP recently dug into its archives to determine whether this stratification is a persistent feature of the entry level law market.  See NALP Bulletin (Jan. 2008).  It turns out that 15 years ago, the market followed a much more traditional distribution.   The chart below summarizes the salaries for the class of 1991.

The 1991 graph is right skewed but bears some resemblance to a normal curve.  Below is the graph for 1996:

The rightward skew is a bit more pronounced and the area under the $75K to $85K range is becoming more substantial.  A more seismic shift is seen in 2000 (below) with the emergence of a second mode at the $125K price point. 

At the height of the Internet boom, a remarkable 14% of all entry level lawyers took jobs at the $125K level.  According to NALP, "never before had a single salary so dominated the landscape."  Under the 2006 bimodal distribution (see chart above), 44% of graduates received entry-level salaries in the $40K to $60K range; yet, the second mode moved further to the right ($135K to $145K) and grew to 17% of all graduates.

What are the market forces that have created this peculiar salary structure?  In my working paper, "Are We Selling Results or Résumés?: The Underexplored Linkage Between Human Resource Systems and Firm-Specific Capital," I posit that the runaway $160K mode is a confluence of two factors:  (1) the continued growth in the corporate legal services market, primarily due to the growing scale and scope of transnational corporate activity; and (2) law firms' nearly universal adherence to the "Cravath system," which purports to hire the best graduates from the best law schools and provide them with the best training.  More after the jump. ...

How has the Part-Time "Loophole" Affected Part- and Full-Time Enrollment? The Data

[UPDATE: An astute reader in comment #1 pointed out that Thomas Cooley's large part-time program could be skewing the data.  That commenter was right.  Thus, I have changed the analysis from the original post.  wdh]

According to Bob Morse's post at Morse Code, law school deans are very unhappy about the proposed change in the U.S. News rankings methodology.  If the changes go into effect, these deans suggest that part-time program will shrink; in the process, students with no other options will be shut out of law school.

There is another way to evaluate this claim.   Is the number of part-time slots propped up by the U.S. News rankings methodology?  In other words, what is the effect of the part-time loophole? 

The table below summarizes changes in 1L full-time and 1L part-time enrollment  between fall 1991 and fall 2006.  (Note that U.S. News debuted its rankings for all law schools in 1992.)  These schools fall into four categories:  (1) schools with FT-only in 1991; (2) schools with FT/PT in 1991; (3) school approved by the ABA-approved since 1991; (4) Thomas Cooley, which has unusually large PT and FT enrollments.  I separate out Cooley because it skews the data (thank you to comment #1 for bringing this pattern to my attention).

Overall, there is a strong tilt toward part-time, but the dynamics are complex.  Here is a summary of the relevant changes (excluding Cooley except where noted):

• Between 1991 and 2006, the number of full-time slots at ABA-approved law school increased by 6.0% (+2,220) while the number of part-time slots increased by a 25.2% (+1,288).  (If Cooley is included, these figures jump to 6.9% and 34.7% respectively.)
• Among the 99 schools with no part-time programs in the pre-ranking era, 14 started part-time programs since 1991 (+497 PT 1L students).  Yet, among the 75 schools that had part-time programs in 1991 (excluding Cooley), the number of part-time slot increased a mere 16 students. 
• Overall, the shift is clearly away from full-time and toward part-time.  Among the 174 schools that were fully ABA-approved in 1991 (again, excluding Cooley), full-time 1L slots fell by 743 students while part-time increased by 513.

Why such a small increase among schools with pre-existing PT program?  There is heightened competition from the new PT programs: during this time period, other law schools added 1,272 more PT seats (excluding Cooley).  Further, schools with higher ranks--and hence more market power--were much more likely to increase their PT enrollments.  Among the 174 schools that were fully ABA-approved in 1991, PT 1L enrollments in Tiers 1-2 climbed by 688 while FT 1L enrollments in declined by 518.  In Tiers 3-4 (excluding Cooley), there was contraction in both FT (-224) and PT (-175) enrollment. 

Let's not kid ourselves. Quite a few students have been shunted into PT programs so a school could drive up its US News numbers.  In my research for an upcoming Southeastern Association of Law Schools (SEALS) presentation, I was shocked by proliferation of so-called "part-time day" programs--which, obviously, do not require professors to teach at night.  These students appear to eventually transfer into the full-time day program; unfortunately, they have to take overloads or summer courses to graduate in three years.    

So here is the bottomline:  Whatever the ranking methodology, a large number of law schools are going to make resource allocation decisions based on rankings.   If  Bob Morse changes the methodology to get rid of the part-time loophole, we can expect many law schools to throw their part-time students overboard--that is a fact, and it is not Bob Morse's fault.   In this US News conundrum, both people and our alleged principles are guaranteed to get hurt.   

The first step toward solving this ongoing embarassment is admitting that we indeed have a problem.  This task should, in theory, fall to the ABA Section on Legal Education and Admission to the Bar, which meets in NYC next month. 

As Andy Morriss and I argued in Measuring Outcomes: Post-Graduation Measures of Success in the U.S. News & World Report Rankings, 83 Ind. L. J. 791 (2008), legal educators can completely eradicate this terrible tail-wagging-the-dog dynamic by releasing school-level data (in spreadsheet format) that reveals detailed information about what students care about: employment by practice setting, salaries, interviews, and bar passage.    Right now, this information is aggregated in ways that impede intelligent decision making by students.   Many schools would be hurt by this approach--but they are not the only constituency that matters.   Students and lawyers are not well served by the opaqueness of the current system. And it is this opaqueness that gives U.S. News rankings so much power.

Empirical Literature on Punitive Damages Cited in the Exxon Case

Today’s Supreme Court decision in the Exxon case relies heavily upon the empirical literature to punitive damages to evaluate the availability and scope of punitive damage awards under maritime law.  See footnote 13 (citing several studies shows that punitive damages were on the rise between the 1960s and 1990s but acknowledging more recent research, including Eisenberg, Heise, et al., Juries, Judges, and Punitive Damages: Empirical Analyses Using the Civil Justice Survey of State Courts 1992, 1996, and 2001 Data, 3 J. OF EMPIRICAL LEGAL STUDIES 263, 278 (2006), which document flat or decreasing punitive damage awards).

After a review of the empirical literature of punitive damages, including the relationship between punitive and compensatory awards, the Court concluded that the real issue it confronted was the "stark unpredictability of punitive awards."  The Court continues, "We are aware of no scholarly work pointing to consistency across punitive awards in cases involving similar claims and circumstances."   There are, however, studies based upon mock juries; the Court dismissed them because they were funded in part by Exxon.  See footnote 17. 

Over at the Election Law Blog, Rick Hasen questions the wisdom of this approach.

[I am familiar with many of the authors and studies cited in footnote 17and it] is really top notch work. So I find this footnote troubling. There will be cases (including election law cases) in which there are no extant studies on an empirical question at the heart of a case. At that point, it makes sense for litigants to fund such research. Indeed, when such research appears in an expert report subject to cross-examination, I assume the Court has no problem relying upon the evidence. So why should it be different when a litigant funds the research, particularly if the research has gone through peer review and of course if the funding source is disclosed so that the opposing side may probe for bias?

Good question.

New York Police Department (NYPD) Stop, Question, and Frisk Database, 2006

ICPSR's recent data releases are online here.  Here is the summary of the New York Police Department (NYPD) Stop, Question, and Frisk Database, 2006, which looked interesting:

These data were originally collected by New York Police Department officers and record information gathered as a result of stop question and frisk (SQF) encounters during 2006. These data were used in a study carried out, under contract to the New York City Police Foundation, by the Rand Corporation's Center on Quality Policing. The release of the study, "Analysis of Racial Disparities in the New York Police Department's Stop, Question, and Frisk Practices" (Rand Document TR-534-NYCPF, 2007) generated interest in making the data available for secondary analysis. This data collection contains information on the officer's reasons for initiating a stop, whether the stop led to a summons or arrest, demographic information for the person stopped, and the suspected criminal behavior.

Transfer Students, Part-time Programs, and US News Rankings: A Response to Ribstein

Larry Ribstein and I are having an exchange on the facts and significance of law school transfer policies.  Following up on my prior post, Larry takes me on over at Ideoblog.  Note that Brian Leiter got the ball rolling yesterday with this post.   Larry is such a formidable intellect that I will surely lose this debate if I remain cornered in his comments section, where I cannot effectively present more data.   Sometimes your own blog comes in handy.

In a nutshell, here is Larry's argument:  I don't have solid evidence that a large number of schools have embraced a class-shrinking policy; further, a .37 gain in a school's median LSAT in exchange for a 10% decline in full-time class size is too big a price for too small a prize [no one had these parameter estimates at the time!].  Hence, the existence of the transfer gaming strategy is speculative.  Since all law schools are trying to be as selective as possible on entry level LSAT scores, why not compensate for some of this underinclusion of qualified people by admitting proven performers as transfers?  If we fully disclose and understand the risks, the costs will be properly internalized by prospective transfer students and law schools.  Under this analysis, Larry surmises there is no obvious right answer on transfer policies.

Larry underestimates the data.  Before reviewing the significance of the statistics in the table below, it is important for readers to understand the basic calculus for gaming the US News LSAT variable:   

• The median for full-time enrollment is the only measure that matters.  PT and transfer statistics are irrelevant.
  
• Most schools are tuition dependent and cannot afford to shrink the 1L class without a countervailing revenue stream.
• One revenue neutral strategy is starting or expanding part-time programs, applying less stringent standards to PT admits, and letting students transfer to FT after the first year.  This strategy, however, is not going to be very effective for a non-urban law school.
• Another is constricting the 1L class (to get the credentials boost) and making up the revenues through more liberal transfer policies.  This strategy, however, only works if a school has plausible lower ranked schools to draw from.  Florida State, with cheap in-state tuition and many lower ranked schools in the state,  fits this profile, though the magnitude of their transfer program (#1 on Leiter list at 24.6%) is pretty shocking.
  

Knowing that shrinking FT 1L enrollment is the lynchpin of the gaming strategy, here is the mean and "left side" of the FT% change distribution--i.e., the schools with lowest growth/largest shrinkage in full-time 1L enrollment.

Note that the data are compiled by quartile because the first full USNWR rankings predated the current "tier" format.  Some observations:  (1) the average change in FT 1L enrollment was positive throughout the law school hierarchy; (2) nonetheless, in each quartile, 25% or more of all law schools shrank their 1L FT class between 1992 and 2004; (3) schools with part-time programs tended to have the largest declines/smallest growth in full-time 1L enrollment, presumably because transfers AND part-time gaming were available to them; (4) the minimum FT% change variable is very low in most cells--why?  I have heard of schools that created "special" summer programs that excluded large numbers of full-time students from their ABA and US News numbers.   If all the surrounding facts became know regarding these programs, it may shock the conscience of most readers, including us lawyers.

The bottomline is this:  64 law schools shrank their 1L full-time entering class between 1992 and 2004, 27 by more than 10%.  At the same time, PT enrollment has boomed and the number of PT programs has grown (by 10 in quartiles 3 and 4).  We don't have transfer data for 1992 to 2004 because the attrition data was only published beginning in 1997, when the ABA-LSAC Official Guide began publishing in a standardized format to respond to pleas from US News that law schools were lying to them.  See Measuring Outcomes, at 11-12.  But the 1997 to 2005 data shows large increases in non-academic 1L attrition, which is the category that includes outbound transfers.

With these trend lines in place, it is easy to see the flaw in Larry's normative defense of liberal transfer policies:   

[Because we have no solid evidence of class shrinkage policy,] the overall conclusion is much more ambiguous than the one Bill draws. The tradeoff would seem to be one of adverse social impact for more opportunity for students who prove themselves to "trade up." With adequate disclosure of the risks, those tradeoffs are internalized by the students and the schools.

Because it is harder to get admitted as a full-time 1L at many law schools (because the entering FT class has shrunk),  these displaced students will be pushed into the transfer market to upgrade the pedigree of their JD degrees.   Sure, we can disclose the risks identified by LSSSE (we could but we generally don't) and presume that our students are all adults and thus can weigh the costs and benefits of transferring.   But collectively, the shrinking supply of 1L FT slots, at least at schools with some regional or national market power, has forced these trade-offs on a larger number of students.  In light of the LSSSE data, more 1L admissions under whole person review (and thus fewer slots for transferring 2Ls) would improve the social networks and law school experience of many students.   Further, the creaming of transfers saps the ability of lower ranked schools to reap the benefits of their innovations.   

Frankly, I don't see any plausible reading of these facts that make students, as a group,  or legal education as a whole (rather than individual, higher ranked law schools) better off.   

Behavioral Economics Primer

I recently ran across this audio file of a talk given at Harvard Law School in 1995 by Charles Munger, entitled "Causes of Human Misjudgment."   Munger is a Harvard Law alum and a founding partner of the L.A. law firm of Munger, Tolles & Olsen.  He later left the firm to run an investment fund.  In the mid-1970s, he joined Berkshire Hathaway to serve as Vice-Chairman with Warren Buffett.

Make no mistake:  Munger is explicitly talking about the intersection of economics and psychology, acknowledging that the nascent (at the time) field of behavioral economics was on the right path and citing insights of Daniel Kahneman and Amos Tversky (note seven years before Kahneman won the Nobel Prize).

I am particularly interested in applied behavioral economics.  And here I don't mean writing papers on the topic;  rather, I mean honing my own decisionmaking processes to eliminate bias and susceptibility to manipulation.  For this purpose, I doubt I will ever find a better resource.  A written version of this talk, including an interesting preface, is online here.   It has quite a few things to say about academia and the pervasive problem of the "truffle hound."  (You will have to read/listen to the links to figure out the truffle hound reference.) 

Finally, Munger's remarks inspired Paul Brest (Hewlett Foundation, former dean at Stanford) and Laura Hamilton Krieger (UC Berkeley) to create a law school  course on professional judgment and decisionmaking at Stanford Law.  The course began in the early 1990s and continues to this day (materials are online at www.professionaljudgment.org).   Several prominent behavioral law & econ scholars took this course while students at Stanford, including Russell Korobkin (UCLA), Jeff Rachlinski (Cornell), and Chris Gurthie (Vanderbilt).    In case there was any doubt, cornerstones of the course appear to be empiricism, applied probability theory, and due attention to disconfirmatory evidence in the face of cognitive biases.  Great stuff.

Storage Solutions for Large Datafiles

Over the years, I have collected a lot of datasets that I have attempted to archive on the university file server.  While computer space is not very expensive these days, it is not free.   These files have, collectively, far exceeded the standard faculty allotment.   

Fortunately, we finally found a simple solution:  The HP Personal Media Drive, which is an external hard-drive with 500GB of memory.  Years worth of files only take up a small fraction of the space.  And just by connecting an USB port, I can quickly backup my entire laptop hard-drive.  It is also very portable.  That is a lot of peace of mind for under $200. 

Large Law Firm Hiring -- Introducing the "Funnel Cloud"

Why do law students follow U.S. News ranking?  In a story in last week's National Law Journal, Andrew Morriss and I claim that USNWR provides a rough guide to future employment prospects.   Yet, drawing upon data on hiring patterns from the NLJ 250 (the 250 largest firms based on number of attorneys) and employment outcomes from the ABA-LSAC Official Guide, Andy and I demonstrate the serious financial perils of overreliance on USNWR.

The story has a lot of data.  The chart below, however, ended up on the editing room floor.  I call it the "funnel cloud."  It shows the percentage of students at each law school that were hired by a NLJ 250 firm.  [Side note: Why is the NLJ 250 a good barometer?  Because the starting salaries make law school tuition look like a bargain.  Based on data made available to us through the Law Firms Working Group,  the weighted average (based on firm size) starting salary at an NLJ250 firm for 2007 was $152,424.]

In my opinion, the above chart has at least two takeaways: (1) the funnel cloud formation shows large law firm employment payoffs are non-linear and that the vast majority of schools offer similarly modest, but not insignificant, entree to this sector; (2) based on the volume of green and red at the top of the chart, most large firms prefer to recruit deeper into the class at a Top 20 school (and will pay a price premium of $160,000 per year) rather than shifting their model to lower ranked schoolss.    53% of all new NLJ250 jobs between 2005 and 2007 went to Top 20 biggest feeder schools from 2005.  See also Leigh Jones, Hiring More Deeply into Top Schools, Nat'l L. J., Apr. 14, 2007.  You would think that by 2007, the Top 20 would have been tapped out.  Not so.

The empirical findings of this 2007 article by Dinovitzer and Garth, which is based on After the JD data, suggest that this elitism has unintended negative consequences in terms of high attrition rates. 

Here are links to other graphics in our recent NLJ story:

Employment trends for law school grads: The top 100 out of 194 ABA-approved schools ranked by percentage of 2005 graduates at NLJ 250 firms.

By Region (defined by states containing Top 10 legal markets):

• New York
• California
• Illinois
• Texas
• Washington, D.C.
• Massachusetts
• Georgia
• Pennsylvania
• Alabama to Kentucky
• Louisiana to Nevada
• Ohio to Wyoming
• Composite report of all 11 regions
• Methodology

New Data: Duration of Capital Appeals

New this week at ICPSR: Multistate Analysis of Time Consumption in Capital Appeals, 1992-2002, by Barry Latzer and James H. G. Cauthen (both of CUNY - John Jay).  The skinny:

Despite public controversy over the length of death penalty appeals, little empirical work has been done on the time allocated to the capital appeals process. The purpose of this study was to perform a multistate empirical analysis of the time expended in direct appeals of capital cases. The researchers included decisions from 14 states that they believed to be representative of the 37 states that have enforceable death penalty laws. For each of the 14 states included in the study, the researchers examined every capital case decided on direct appeal by the courts of last resort between the dates January 1, 1992, and December 31, 2002. The researchers developed a case database by examining a variety of sources. For each of the 1,676 cases in the multistate database, the research team collected time consumption data for each of the following five phases of the direct appeal process: (1) the postsentence stage, (2) the preparation stage, (3) the argument stage, (4) the decision stage, and (5) the supreme court stage. Variables include state, case characteristics, court opinion variables, dates, and time consumption variables.

The states are Arizona, Florida, Georgia, Kentucky, Missouri, Nevada, New Jersey, North Carolina, Ohio, South Carolina, Tennessee, Texas, Virginia, and Washington.

New Release: UCR Crime Data

Over on his blog, Andrew Gelman notes the release of a massive new dataset on crime:

"The time series in question are monthly data from 1960-2004, for over 17,000 police departments, for seven crime types (murder, rape, robbery, aggravated assault, burglary, larceny, vehicle theft), as well as their sum (the so-called Crime Index), and an additional 19 subcategories – e.g., robbery with a gun, knife, personal weapons (hands, feet, etc.), or other; attempted rape; auto, truck or bus, or other vehicle theft...The data consist of monthly counts of these crimes reported by police departments throughout the country to the FBI as part of its Uniform Crime Reporting (UCR) Program."

The data are being made available by Michael Maltz, at Ohio State's Criminal Justice Research Center.  Get them at http://sociology.osu.edu/mdm/UCR1960-2004.zip.  Maltz also makes some great points about the reliability and validity of self-reported crime data; go check it out.

New ICPSR Data Releases

For Monday, January 6, 2008.  Here.

Hoffman on Empiricism and Corporate Law Research Agendas

I have spent the last several days at the AALS Conference in New York City listening to some interesting panels and having lots of interesting conversations with colleagues from other law schools or law-related institutions--stuff that impacts both my teaching and scholarship (and what I will be blogging about in 2008).

If anyone doubts the substantive value of attending AALS, check out David Hoffman's post on paths for corporate law scholarship, which flows from his conversations with professors from other law schools.  Dave outlines six potential paths, and five draw on some form of empiricism.  Here is an excerpt.

I've had a few conversations at the AALS this year revolving around the same topic: where is corporate law scholarship going? These conversations are naturally framed by being a (still relatively) junior scholar, not wanting to pick a research agenda or field that will be seen to be a dead-end in a few year's time. Over these conversations, folks have suggested several different types of corporate law research agendas. Obviously, the categories overlap and over-simplify, but it seemed useful to me to list them here, with a few thoughts on the obvious promises and perils of each.

1.  Market Quant-heads: Using Tobin's Q and other quantitative methods to evaluate governance.  Pros:  still lots of low-hanging fruit, interesting results, multiple papers per data set.  Cons: some barriers to entry if you aren't afraid of being publicly wrong, severe barriers to entry if you are; co-authorship with finance folks is useful but quality proxies are unreliable; probably little connection with your day-to-day teaching responsibilities; unclear value-added over business school professors; not clear how easy it is to place this work in flagship mainstream law reviews, if your law school values this currency (which it shouldn't).

2. Contract Quant-heads:  Evaluating databases of agreements to shed light on private law-in-action.  Pros: great work being done in entrepreneurial law, and huge areas of private corporations and LLCs are still basically unexplored; the learning curve, once you are familiar with STATA, is not steep; lawyer competencies are essential to the project. Cons: coding takes time, and projects can take >18 months to complete; finding appropriate data sources requires connections you likely do not have at the beginning of your career; you will almost certainly eventually want to get a grant, which creates its own set of bureaucratic issues.

3. Doctrinalists:  Careful reading of Delaware cases, often informed by some L&E or BL&E theory.  Pros: there are always new cases to write about; consulting gigs abound; lots of tie-in with teaching; influence over development of law; doctrine probably still places pretty well. Cons: in a globalizing economy, Delaware may be a declining stock to ride; elite law schools will likely not hire you; the "grand theory" field is very rich and possibly occupied.

Dave also discusses modelers and historical analysis, experimental design, and  qualitative comparative governance.  Very thoughtful stuff.