In my post on Saturday, I argued that John Lott’s complaint
against Steven Levitt defines replication too narrowly. According to the complaint, Levitt’s reference to replication in Freakonomics can mean only one thing: analyzing “identical data” to Lott’s data and analyzing it in the same way that Lott analyzed it. (Compl. ¶ 12.) Again, this meaning of replication is valid. It matches what Paul Sniderman
described as the first of three common definitions of replication, the
verification of reported results using the same data and the same analysis. See Paul Sniderman, “Evaluation Standards for a Slow-Moving
Science,” 28 PS: Political Science and Politics 464, 464 (1995). But as before, the point is that there are other valid meanings of replication. Last time, my examples were based primarily on the writings of political scientists. This time, I'll provide examples from Lott’s own writings, which I found after posting my initial comments.
The first example comes from Lott’s article examining claims of racial discrimination at the polls in the 2000 election in Florida. For present purposes, the relevant material is in the appendix, which is labeled, “Using the Majority Report’s County-Level Data.” What he is referring to is the U.S. Commission on Civil Rights’ report, Voting Irregularities in Florida During the 2000 Presidential Election (2001). It’s a “majority” report because two members of the commission dissented. Here is what Lott says about the report:
Since the commission’s report, which was presented to the Senate, shows exactly what regression specifications were examined, I tried different specifications to replicate the commission’s results. The results (available from the author) showed that the simple cross-sectional county-level data were very sensitive to the specification used and that most specifications failed to show any statistically significant positive relationship between the percentage of voters who were African-American and the non-voted ballot rate. Thus, when the commission’s very own set of control variables is used, there is thus no real confidence that there is a positive relationship between the share of African-American voters and the ballot spoilage rate.
John R. Lott, Jr., “Nonvoted Ballots and Discrimination in
The second example is from Lott’s point-by-point response to a critical column by syndicated columnist Michelle Malkin. In it, he references replication twice.
I spent years replacing the data lost in the hard disk crash. The county level crime data was replaced and given out to academics at dozens of universities so that they could replicate every single regression in More Guns, Less Crime. I have also made the data for my other book The Bias Against Guns available at https://www.johnlott.org/cgi-bin/login.cgi . The data for my other reserach [sic] has also been made available. The survey was also replicated and obtained similar results to the first survey and the new data has been made available since the beginning of the year.
https://johnrlott.tripod.com/malkinsoped.html (emphasis added but note that the original version is entirely in italics). The first reference to replication in this paragraph matches Sniderman’s first definition, the verification of reported results. The second reference matches Sniderman’s third definition, “the use of a different data set and comparable measurement and estimation procedures, to validate the robustness of both the results initially observed and the interpretation originally given to them.”
Replication is a convenient term for describing several related activities. Even Lott’s writings demonstrate this.