While independently replicating each others results (just to be safe and prudent), a colleague and I inadvertently uncovered an issue whose import I had previously (but no more) underestimated--stats software updates.
In our specific instance, Stata's move from version 9.0 to 9.1 (I now use 9.2) implicated, among other commands, those germane to clustered standard errors. One result of Stata's update was ever-so-slightly different output (again, only relating to our clustered standard errors). Because we were working off the identical data and do files (but, alas, as we subsequently realized, slightly different Stata versions) any discrepancy alarmed us. An incidental hallway chat with Ted Eisenberg reminded me of Stata's version control command which allows users to run prior Stata versions. Once I did, exact replication (and much relief) emerged. (And for purposes of this discussion, let's set aside disputes about what "replication" might mean in a legal sense.) Because I frequently flip between stats packages (e.g., SPSS and Stata), I've always been mindful of subtle differences between stats software packages. This episode taught me that I need to be similarly sensitive to subtle differences across versions within a software package.
To the extent that this is an issue that will likely persist over time, perhaps we need to develop a norm promoting the disclosure of the software version used to generate results so that those following our footsteps can save some time.
I recall years ago that SPSS used to add 5.0 to the constant of probit equations. This was before maximum likelihood procedures were perfected, and there was apparently some value in keeping the cumulative normal distribution above 0. But if that was the reason, why they couldn't subtract the 5.0 after the calculations was never clear. Needless to say, folks who didn't RTFM and didn't notice this undoubtedly made massive mistakes.
Oh yeah, for logit models, they added 5 to the constant and then divided other coefficients by 2 "in order to make them comparable to probit coefficients."
Posted by: Jeffrey Segal | June 16, 2006 at 07:26 PM
Good point, Michael. And, as Jeff implies, version differences are often a less significant issue than differences between packages. For interested people, a nice resource that covers this issue and a host of others is Altman, Gill, and McDonald's (2003) "Numerical Issues in Statistical Computing for the Social Scientist" (John Wiley & Sons). A couple sample chapters and some other related materials are available here:
http://www.hmdc.harvard.edu/micah_altman/numal/
Among other things, they show how, thanks to issues with numerical precision and formulae, a number of software packages (including Excel!) can't even calculate a standard deviation correctly. (For a hint at possible implications of this for Sarbanes-Oxley enforcement, see http://hardware.slashdot.org/article.pl?sid=06/06/05/185201 ). Sobering stuff.
Posted by: Chris Zorn | June 16, 2006 at 07:29 PM