Last week, Lott filed his opposition to Levitt’s and HarperCollins’ motions to dismiss and explained his argument about the meaning of replication. Recall that Lott claims as follows: “‘replicate’ has an objective and factual meaning in the world of academic research and scholarship.... the clear and unambiguous meaning is that ‘other scholars’ have analyzed the identical data that Lott analyzed and analyzed it the way Lott did in order to determine whether they can reach the same result.” (Complaint ¶ 12.) The key point here is that replication means only one thing, using the identical data and analyzing it in the same way as the original analysis. It’s replication in the sense of verifying the original results. It does not involve a new sample. It does not involve a new model.
Although Lott could have made a strong case for the
Lott’s claim is limited to one for defamation per se. According to the complaint, the passage in Freakonomics “is defamatory per se because it attacks Lott’s integrity and honesty in his profession as an economist, scholar and researcher.” (Complaint ¶ 14.) As a claim for defamation per se rather than per quod, the statement must be “obviously and materially harmful” on its face without resorting to extrinsic facts.
The defendants claim Lott is introducing extrinsic facts
insofar as he is explaining the meaning of replication in “the world of
academic research and scholarship.” Nothing in the defendants’ argument really
persuades me that explaining the meaning of a word qualifies as an “extrinsic
fact” for defamation purposes. Suppose the allegedly defamatory statement was
in Spanish. Would the explanation of what the statement means in English count
as an extrinsic fact? It seems unlikely. In one Northern District of Illinois
case (though one controlled by
The Excerpt is not defamatory per se because the words are not sufficiently harmful on their face. Although Plaintiff claims that the statement that “other authors have tried to replicate his results” accuses him of falsifying his results, Compl. ¶ 12, it is only by explaining that “the term ‘replicate’ has an objective and factual meaning in the world of academic research and scholarship,” and that as applied here, it means that “‘other scholars’ have analyzed the identical data that Lott analyzed and analyzed it the way Lott did,” that he arrives at that conclusion.
Id. These are extrinsic facts, and a statement that requires extrinsic facts to explain its defamatory meaning cannot be defamatory per se.
(HarperCollins Memorandum at 12-13.) But the word replication must mean something on its own. The defendants explain what it means in this way:
The authors [of Freakonomics] ... report that some “other scholars” have drawn different conclusions from Plaintiff, because those scholars could not substantiate (or “replicate”) Plaintiff’s results. The Excerpt clearly does not say that these other scholars (who are specifically identified in the Notes at the back of the Book) followed precisely the protocols used by Plaintiff. The Excerpt, fairly read, says only that the conclusions reached by these scholars did not support Plaintiff’s “hypothesis”, a meaning reinforced by the remainder of that sentence: “they found that right-to-carry laws simply don’t bring down crime.
(HarperCollins’ Memorandum at 9) (emphasis in original).
In response, Lott refers to several dictionary definitions, claiming that “the Court need go no further than the dictionary definitions ... to conclude that Defendants’ statement is defamatory per se. Those definitions state the same meaning of replication that is set forth in the complaint.” (Lott’s Opposition at 19.)
Even assuming these dictionary definitions are adequate indicators of the meaning of replication in academic work, they are not especially helpful to Lott. His lead example comes from Webster’s New World College Dictionary (4th ed. 2004). According to Lott, the relevant definition of replicate in this dictionary is “to repeat, duplicate, copy, or reproduce.” And the relevant definition of replication is: “Statistics the exact duplication of an experiment for verification, criticism, or extension of previous results.” (Lott’s Opposition at 11.)
When an experiment is repeated or duplicated, however, the data is unlikely to be identical to the original data. Sampling and measurement error often introduce differences, sometimes small and sometimes large. Moreover, the reference to the “extension of previous results” in the second definition indicates the use of new data. An analysis of some feature of the Seventh Circuit, for example, might be replicated or repeated with data from the First, Ninth, and D.C. Circuits. The data from these other circuits will not, of course, be identical to the original data from the Seventh Circuit. If the same result is found in these other circuits, then we can fairly speak of extending the original results, which were based on the Seventh Circuit. In the definition of replication in a social science dictionary, it says, “Repeating studies with different subjects or in different settings is especially important for experimental laboratory research, because it helps increase external validity.” W. Paul Vogt, Dictionary of Statistics & Methodology 275 (Sage 2005). Repetition does not necessarily mean everything stays the same. The bottom line is that these definitions in Webster’s are not equivalent to the narrow definition in the complaint.
Of the several dictionaries Lott quotes, the MSN Encarta definition of replicate is the closest to what Lott wants. It refers to making “an identical version of something repeatedly and exactly.” It also refers to doing “something again in exactly the same way.” (Lott’s Opposition at 11.) But doing something again, even in “exactly” the same way, could fairly include the use of the same methodology to generate a new random sample. A new sample means new and, most likely, non-identical data. (Also, compare Encarta’s definition of replicate to its definition of replication. The latter does not use the word identical or the phrase exactly the same way.)
Lott also cites three examples from academic sources. While one appears to use replicate in the narrow sense of verification, which is Lott’s definition, it does not show that this narrow sense is the only common and obvious use of the term. It may, however, serve as a response to the defendants’ argument, express or implied, that replication does not, on its own, refer to using the same data and analysis. The other two sources are not cited in enough detail, so I can’t tell how the authors are using the term (and I have not looked them up).
Assuming these academic authors are using replication in the Lottian sense, what of it? Lott’s definition of replication is, as I have said before, consistent with common usage. The problem for Lott is that other definitions are also consistent with common and fair constructions of the word. But even if Lott is right about the meaning of replication, I don’t see how the statement in Freakonomics is defamatory. If someone can’t replicate your results in the narrow sense, it does not inevitably mean you falsified your results. It could just mean you made a mistake, which happens quite often in quantitative work because it is easy to make them. See David Salsburg, The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century 85-86 (Henry Holt 2001) (“Very few mathematicians can work alone.... You need to expose your new ideas to the criticisms of others. It is so easy to make mistakes or to include hidden assumptions that you do not see, but that are obvious to others.”).