Lotts of Replication Standards?

As Michael noted yesterday, one of the key issues in Lott v. Levitt, No. 06C 2007 (N.D. Ill. 2006), is the meaning of the term replicate or replication in academic scholarship. In the first of two defamation counts, Lott alleges that a statement on pages 133-134 of Freakonomics is false and defamatory, an argument tied to a particular meaning of replication. (Compl. ¶¶ 9, 11, 14.) Lott claims it has an “objective” meaning at odds with Levitt’s use of the term. (Compl. ¶ 12.)  At Overlawyered, Ted Frank suggests that Levitt’s defense may depend upon persuading the court that he was referring to replication in the lay sense, not the technical sense. The technical meaning of replication, however, is not settled.

According to the complaint:

The term “replicate” has an objective and factual meaning in the world of academic research and scholarship. When Levitt and Dubner allege that “other scholars have tried to replicate his results,” the clear and unambiguous meaning is that “other scholars” have analyzed the identical data that Lott analyzed and analyzed it the way Lott did in order to determine whether they can reach the same result.

(Compl. ¶ 12) (emphasis added).

Lott’s conception of replication is very narrow. One of his requirements for replication is the use of the exact same data. Apparently, the other requirement is the use of the exact same method of analysis. Paragraph 11 supports this interpretation of the second requirement. It says, “no scholar who has replicated Lott's statistical analysis has concluded that the data and methods on which he relied don’t support his conclusion.” (emphasis added). These two requirements, using the same data and using the same analysis, match one common definition of replication, but its meaning in the academic world is not clear and unambiguous. There are other common meanings of replication.

The September 1995 issue of PS: Political Science and Politics, which included a symposium on replication with contributions from nineteen political scienists, illustrates this point. The title of the symposium was “Verification/Replication,” which brought to the fore the problem of terminology. According to Robert Hauck’s introduction to the symposium, the minimum goal of the exchange was to achieve some common understanding of several terms, including verification, replication, secondary-analysis, and cross-validation. Robert J-P. Hauck, “Oh Monsieur Pasteur, We Hardly Knew You!” 28 PS: Political Science and Politics 443, 443 (1995). Unfortunately, the symposium participants could not achieve even this goal. Hauck noted that until everyone could settle on a shared vocabulary, “symposium readers are cautioned to remain alert to the many meanings of replication and its alternatives.” Id.

Gary King’s lead article in the symposium proposed a “replication standard” to promote the full disclosure of data sets and methods so that other researchers could “understand, evaluate, and build upon a prior work.” Gary King, “Replication, Replication,” 28 PS: Political Science and Politics 444, 444 (1995). In King’s view, this standard does not necessarily require sharing data, but he suggested the standard could rarely be met in practice without sharing them. Id. at 445. Either way, the goal of this standard is to make it feasible for scholars to understand and verify each other’s results.

While some participants in the symposium found no problem with King’s use of the term replication for what he was describing, other scholars challenged it, arguing King was really talking about verification. Paul Herrnson said King “mistate[d] the meaning of replication in the physical and life sciences, as well as political science.” Paul S. Herrnson, “Replication, Verification, Secondary Analysis, and Data Collection in Political Science,” 28 PS: Political Science and Politics 452, 452 (1995). Joel Aberbach and Bert Rockman agreed with Herrnson: “[A]s with Herrnson, we are puzzled by all the fuss over what he notes is not really replication, but merely verification.” Joel D. Aberbah and Bert A. Rockman, “Engaging the Periphery--More Busywork for Political Scientists,” 28 PS: Political Science and Politics 476, 476 (1995).

In addition to calling verification what Lott calls replication, Hernnson rejected the need to use identical data for replication. Replication in Herrnson’s view means completely repeating a study, including the data collection process. Unless you use the original data set, however, differences between the original data and the replicated data are likely, but on this view of replication, the goal is not necessarily the exact duplication of the original data set. Replication “enables a researcher to comment on whether data used in an original study were collected properly or whether generalizations supported under one set of conditions are also supported under others.” Herrnson at 452 (emphasis added).

Avoiding the semantic disagreement over the meaning of verification and replication, Paul Sniderman simply recognized that replication is used and understood in different ways. He described three meanings of the term:

Replication in sense 1 involves the use of the same data set, procedures of measurement, and methods of estimation to verify the accuracy of reported results. Replication in sense 2 involves the use of the same data, but not the same methods of measurement or estimation, to confirm the adequacy of the interpretation [and] reported results. Replication in sense 3 involves the use of a different data set and comparable measurement and estimation procedures, to validate the robustness of both the results initially observed and the interpretation originally given to them.

Paul Sniderman, “Evaluation Standards for a Slow-Moving Science,” 28 PS: Political Science and Politics 464, 464 (1995).   

As Sniderman noted, King focused on replication in the first sense, verifying the accuracy of reported results. Clearly, however, other meanings of the term are in play. And despite King’s focus on replication in terms of verifying results, which is Lott’s form of replication, King did not claim this meaning is the only valid one. It was simply the one of primary interest in King’s proposal. King made it clear that he recognizes other meanings of the term when he summarized the following question and then offered his response:

Why are we worrying ourselves with what might be called “duplication” of existing research?  Isn’t the more important question actual replication where the same measurements are applied to new substantive areas, countries, or time periods?

Good science requires that we be able to reproduce existing numerical results, and that other scholars be able to show how substantive findings change as we apply the same methods in new contexts.... In fact, we can encourage scholars to pursue replication in new contexts if they can be more certain of present results.

King at 451 (last emphasis added). Not only does King add yet another term to the mix, duplication, but he also recognizes that replication can involve new, non-identical data.

Lott’s claim about the one true sense of replication is inconsistent with this discussion among leading scholars. There may not be many ideas about the meaning of replication, but there are several, at least among political scientists. The same is probably true in other disciplines, where different meanings of the term are also used and recognized. See, e.g., Robert Rosenthal, “Replication in Behavioral Research,” in Replication Research in the Social Sciences 1, 5 (James W. Neuliep ed., 1991) (“If we choose our replications to be as similar as possible to the study being replicated, we may be more true to the original idea of replication but we also pay a price; that price is external validity.”); Louise H. Kidder and Charles M. Judd with Eliot R. Smith, Research Methods in Social Relationships 26 (1986) (“Replication means that other researchers in other settings with different samples attempt to reproduce the research as closely as possible.”).

Comments

[William Henderson]

Bill,

Your blog entry may be cited in Levitt's (successful) motion for summary judgment. Very interesting. bh.