More on Grades and Success in Big Firms (Sander guest-entry 3)

I've been slow to post a third entry this week because some of the comments on my first entry made me think more carefully about the table I posted Monday (entitled "White Michigan Grades at Private Firms with 100+ Lawyers) and the conclusions I drew from it.  I went back to the data and offer below a refined analysis.

There are two significant weaknesses in Monday's table. First, I included all Michigan alumni in creating the deciles, but of course many alumni did not return surveys. These non-respondents are part of the "base" on which the rates are calculated. So the author of the first comment, who noted that the proportion of Michigan grades joining firms seemed way to low, had a good point. (Among respondents, about 37% started at firms with 100+ lawyers and 47% started at firms with 50+ lawyers. The numbers would probably be higher today because firms are bigger.) Second, the time periods covered by the first two columns and the third column are significantly different; the first column looks at graduates from 1977 to 1990, while the third (15-year column) looks at grads from 1967 to 1985. Even though I noted the data were not literal attrition rates, many readers took them to be that.

After some thought about these limitations, I concluded that a better way to look at the association between GPA and success at big firms was to look directly at survival rates for big firm associates. The Michigan alumni surveys ask respondents how long they stayed at their first job after graduation, and how many jobs they've had. It's thus possible to see how many grads who started at big firms stayed at the same firm for the next fifteen years (and, in nearly all cases, became a partner at the firm).

The table below summarizes such an analysis. Those tracked here were respondents to the 15-year alumni survey from 1987 to 2000 who reported that their first job after law school was at a firm with 50 or more attorneys; these are divided into grade deciles, and tracked by whether they are still at the same firm. (I used a 50-attorney threshold because many of these attorneys started their careers in the 1970s, when a 50-attorney firm was considered "big".)

(a) Grade Decile	(b) White respondents to 15-year survey who started at firms with 50+ lawyers	(c) Those in (b) who were still with original firm 15 years later	(d) Those in (b) no longer at original firm	% Still at Original Firm (c/b)
1	25	1	24	4%
2	32	3	29	9%
3	58	7	51	12%
4	62	12	50	19%
5	94	13	81	14%
6	136	33	103	24%
7	157	26	131	17%
8	207	55	152	27%
9	236	56	180	24%
10	185	60	125	32%
Total	1192	266	926	22.3%

To illustrate how to read this table, look at the first row of data.  Of 1192 Michigan 15-year alumni who provided information on their first job and who started their career at a big firm, 25 (2%) had grades that put them in the lowest decile of all Michigan students.  Of those 25, only 1 was at the same firm 15 years later - a 4% "survival" rate (reported in the last column, as (c/b).

The last column thus gives us an idea of how grades affect tenure among those at big firms.  The relationship is quite strong -- the correlation of grade decile and the "survival" rate is more than .90.

With this added context, let me make some observations responsive to the posted comments and other emails I've received:

1) I don't contend that high law school grades themselves have a direct causal effect on success in big firms. I contend that high grades measure various things that are related to later success. Those things might conceivably include actual learning in various law classes; more likely, they include facility with legal analysis, cognitive skills and writing ability, and also such things as ambition, drive, lack of a social life, etc.

2) Obviously another factor that affects "survival rates" to partnership are choices about career, family, and lifestyle. Those choices are certainly very different for women and men, and they might vary across racial lines, too. But in The Racial Paradox, I showed that during the early years of associate experience (the years for which the After the JD project supplies good data), white men and white women had essentially identical experiences, while those of blacks and Hispanics diverged dramatically. So for women, attrition due to personal choices or to "glass ceilings" presumably sets in later.

3) Several of those posting comments suggested that minority patterns of attrition must be far higher than those explained by these grade variations. The Michigan data can't really answer this question, because the samples of nonwhites in big firms are very small (a total of 43 nonwhites in the 15-year alumni dataset reported starting at big firms). The numbers we do have for UMLS nonwhites are roughly consistent with those for whites:

(a) Grade decile range	(b) Nonwhite respondents to 15-year survey starting at firms with 50+ lawyers	Proportion of (b) at same firm 15 years later	Comparable rate for whites
1-3	30	10%	10%
4-6	9	22%	20%
7 or higher	4	25%	25%

Again, the sample sizes are far too low to draw conclusions from this second table; the close match of percentages is largely happenstance. And of course the dynamics facing nonwhites are very different. As I observed before, the nonwhites with low grades face two extra obstacles: first, they are likely to be at more elite and challenging big firms than are the whites with low grades; second, they are likely to be "stereotyped" as second-rate even when they are not. On the other hand, minorities may benefit from some affirmative action at the promotion and partnership level -- though clearly, if this happens, it is much less than the use of racial preferences in the hiring of associates.

As this second table suggests, nonwhite hires at big firms have far lower GPAs than white hires; nearly two-thirds of the nonwhites are in the lowest two deciles, compared to 5% of the whites.

The story told in these short blog entries is a brief summary of a more complicated story laid out in The Racial Paradox. I've tried here to grapple with the responses of those skeptical of the data.  My hope (perhaps a forlorn one) is that critics will go beyond mere skepticism to try and develop an alternative explanation consistent with all of the available data.

Comments

[Rick Lempert]

Professor Sander's analysis in this blog is obviously a first cut with the data he is using, as evidenced by the fact that it revises and improves upon an analysis reported in his initial blog in this series. While the data, on whites especially, seem strongly supportive of the Sander hypothesis that law school grades reflect traits essential to success in large law firms, there is in fact substantial ambiguity in the data as reported. This ambiguity will hopefully be resolved through deeper analyses.

First, the Michigan alumni data set that is the basis for the table aggregates data from a successive cohorts of Michigan alumni surveyed 15 years after graduation. However, for Prof. Sander's purposes, it is essential that cohort graduation year be controlled. This is because, if my recollection of the changing marketplace for Michigan graduates is correct, in the earlier part of the time series large law firm recruitment was concentrated toward the top of the Michigan graduating class and at that time (e.g. my graduation year of 1968) the chances of making partner at the firm one started with were reasonably good and known within 7 years. As the years passed, large firms began to recruit from further down in the class and to make partnerships more difficult to achieve. Moreover, in more recent years law firms have done much more lateral hiring of partners, meaning that a partnership in the firm one started with relates less strongly to one's performance than it once did. Thus a good part of the correlation between law school class rank and original firm persistence (meaning in most cases partnership) may be spurious, a result of the fact that large firms began recruiting more people with lower class ranks during the years when partnership promotions became less likely and lateral moves to partnerships or positions in other firms increased.

Also time sensitive is the definition of large firm employment. While firms of 50 attorneys might be considered large in the early years of the Michigan time series, by the later years firms of this size were considered boutiques. Rather than define large firms with a constant number, changes in firm size might be better captured if for each cohort Professor Sander took the firm size that encompassed the median Michigan graduate (with a minimum of 50 attorneys in the early years) and considered any graduate in a firm of that size or larger to be in a large firm. The measure cannot be ideal because, as I recall, the alumni survey does not report numbers of attorneys but just ranges. Still defining "large firm" by cohort seems a better strategy than choosing a figure that is numerically constant but changes in meaning, such that it no longer defines a large firm, over time.

Equally important is a control for gender, as women seem less likely to remain in large firm practice than men for reasons that most likely have nothing to do with different skill levels. (Prof. Sander's second blog posting which shows women distributed more or less like men in the AJD data set may or may not be a cause for optimism since the situation after a decade may be quite different from what it is after two years. He might crudely check on the likelihood of this possibility by looking at how dissimilar initial positions for men and women in the Alumni data set are after 5 years and compare this with position dissimilarity in the same cohorts after 15 years. Similar levels of dissimilarity would suggest that initial distributions persist. An increase in dissimilarity would suggest that not too much should be made of initially similar job distributions.) The gender control is particularly important if one is seeking to get a handle on minority performance as a far higher proportion of minorities than whites were women. (Over the 1970-1996 time period about 37% of minorities were women as opposed to about 24% of whites.) Since most blacks have relatively low grades compared to Michigan's white alumni, in an uncontrolled analysis gender bias in large law firm drop out could be mistakenly attributed to grades.

Anther helpful adjustment would be to exclude from the base those who began their careers in large firms but are currently in teaching, judgeships, elective office or (possibly) general counsel or associate general counsel positions with large companies. Large firm practice is often a stepping stone to the first 3 of these. While the fourth was at one time a consolation prize for those who didn't make partner in large firms, it was also suggestive of considerable accomplishment since firms would not cooperate to place their associates who did not make partner into the general counsel's offices of their clients if they did not think these lawyers would reflect well on the firm and maintain close relations with the firm. Thus if grades are hypothesized to relate strongly to whatever it takes to be a success in an elite law firm practice, one should not include as negative cases those whose career paths from large firms suggest either that they were reasonably successful within them or that they were using the large firm as a jumping off point for different careers. Certainly no one would assume that those who left firms to enter teaching had poorer legal analytic skills than those who didn't make this move, and movements to judicial or political careers have such attractions for certain people that no inference of an inability to succeed in the large firm context should be made from making this move. The latter is particularly important in work that seeks to examine the success of minorities in large firms, for the visibility of minorities employed by large firms often makes them prime candidates for movement to these kinds of prestigious public service.

I have never worked closely with the Michigan Alumni Data set but I thought it contained information on current partnership status and firm size. If so rather than looking at original large firm persistence after 15 years as it relates to class rank, one can look at those who started in one large firm and 15 years later are partners at that firm or another large firm. (Note that for reasons indicated above, the definition of large firm need not and probably should not be the same for original and current position.) In a world where law firm mergers and the lateral hiring of partners has become more common, any large firm partnership after 15 years seems a better measure of success in large firm practice than a partnership in the original firm after 15 years. Indeed, since one can persist in an original firm as a part timer or associate, the equivalent measure would be any membership in a large firm after 15 years.

Finally Professor Sander is correct that there are too few blacks in the UM alumni data set who began in large firms to make much of the grade/partnership correlation in just th black population. This is even truer when one applies necessary cohort and gender controls. Thus I doubt if the table on black grades and law firm performance that Prof. Sander presents in his blog above is worth presenting. But since he has presented it, I would ask that he present the information for blacks as he does the information for whites, by all deciles and let readers make their own aggregations. If an N of 4 is sufficient for the top 4 deciles and an N of 9 for the middle 3, surely the N of 30 for the lowest three deciles is sufficient to allow the disaggregation of these deciles. Since it is in the bottom 3 deciles where blacks hired by large firms are concentrated, it would be interesting to know if within the ranks of blacks hired by large firms, there was a strong relationship between law school grades by decile and persistence to partnership. (One might also note that any judgments of the relationship between grades and advancement to partnership for those with low grades is made more difficult by selection bias. For example, whites hired by large firms in the lowest deciles may be hired because they are the children of partners in other large firms where they cannot be hired because of nepotism rules. Apart from this they may be weaker than even their grades suggest, or if they succeed their parent's firm may be willing to bring them in at the partner level although they would not have hired them initially. Blacks with low grades could, conversely, have aspects to their records that suggest they will be stronger performers than their grades indicate.)

One final comment for the moderators as much as anyone. I note that the prior commentator on this blog and most on Prof. Sander's first blog, use pseudonyms or first names. I don't think this kind of anonymity has any place in an academic blog like this one, except in rare circumstances that should be approved by the moderators on an as needed basis. I don't think people should be able to criticize academic work while holding their own identities secret, and I know that I might be helped in evaluating criticism and comments by knowing who the critic or commentator is.

Rick Lempert

[Captain Obvious]

As I observed before, the nonwhites with low grades face two extra obstacles: (1) they are likely to be at more elite and challenging big firms than are the whites with low grades; (2) they are likely to be "stereotyped" as second-rate even when they are not.

The reason for much of the skepticism is your lack of substantiation for (1); your presumption that (1) significantly outweighs (2); and your failure to investigate (2) with any rigor or attention even coming close to your unhealthy obsession with (1).