« More Methodological Tools | Main | More on IRBs »

16 December 2006


More info please

Can you provide a sense of the effect sizes for the differences? P values can only tell us so much.


Perhaps there's a selection bias in associates working in NYC versus those working in the other cities. How an associate working in NYC rates say, Training and Guidance, is different from an associate in Boston rating Training and Guidance. Thus rendering the observations across various cities not comparable, even in aggregate form...

Mike Guttentag

A couple of questions/thoughts. First, I see no reason to expect that the “Overall Workplace Rating” needs to be defined by the seven topics that the survey also included. You mention future compensation, but what about present compensation? Also, it would seem that one’s attitude would be shaped by the expected likelihood that you would make partner, and I would suspect this varies systematically depending on whether you are at a main office or regional offices, which, in turn, may be associated with the size of the legal market.

Another issue that certainly complicates all of this analysis is a self-selection bias. Initially, I thought this would tend to lead your results to underestimate the actual effect size. If only those willing to tolerate an unpleasant environment went to New York, say, then it would have to be really awful to exceed their expectations for unpleasantness. On the other hand, we are talking about individuals who choose to work for law firms in New York. Maybe their self-reporting of their work environment is negatively biased, as these may be the type of hyper-competitive people who are never satisfied with the status quo. Thoughts from someone whose legal work experience consists of some halcyon summers as an over-paid, under-worked summer associate.

William Henderson


You raise some fair points. A few responses:

First, the ALM rankings of firms is not very instructive, primarily because it is the product of weighted and aggregated inputs, just like US News. It is the inputs that are relevant here, not some formula announced by Am Law. Further, this is a very competitive market for associates, so the work conditions in the broad middle should be (and are) very similar. Ranking them magnifies tiny changes among very similar firms. And unlike US News, law firms cannot control what they report.

Second, the response rates are not that bad: an average of 44 percent in 2004 and 42 percent in 2005.

Third, I am weighting/aggregating specific input scores of branch offices in particular markets. These are, in essence, average of over several hundred 3-5 year associates working in those markets.

Fourth, the metro-level analysis reveals a remarkably consistent pattern from NYC to the large market to the middle markets. It is highly unlikely that response rates are going to produce this pattern.


Bill Ford


The annual changes in the American Lawyer rankings makes me wonder how good the data are for comparing individual law firms.

I took a quick look at the 2004 and 2003 midlevel associate rankings, and for the 125 firms ranked in both years, the average change for individual firms is about 29 ranks.

For example, the biggest move from 2003 to 2004 involved Chapman & Cutler. It was ranked 145th out of 159 firms in 2003, but it was ranked 21st out of 149 firms in 2004. Many other firms also moved way up or way down.

It seems unlikely that law firm policies and cultures are really changing enough from year to year to explain these variations. More likely, the high turnover rates at many law firms and the low response rates produce the extreme variations from year to year.

Now, you are aggregating the firms together from various regions, which might overcome the problem. For comparing individual firms, the rankings might mean very little, but across many firms, meaningful patterns for the various regions might emerge.



AmLaw Links: http://www.law.com/jsp/tal/PubArticleTAL.jsp?id=1096473926557

The comments to this entry are closed.


July 2024

Sun Mon Tue Wed Thu Fri Sat
  1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30 31      

Site Meter

Creative Commons License

  • Creative Commons License
Blog powered by Typepad