More on Lott’s Replication Standard(s)

In my post on Saturday, I argued that John Lott’s complaint against Steven Levitt defines replication too narrowly. According to the complaint, Levitt’s reference to replication in Freakonomics can mean only one thing: analyzing “identical data” to Lott’s data and analyzing it in the same way that Lott analyzed it. (Compl. ¶ 12.) Again, this meaning of replication is valid. It matches what Paul Sniderman described as the first of three common definitions of replication, the verification of reported results using the same data and the same analysis. See Paul Sniderman, “Evaluation Standards for a Slow-Moving Science,” 28 PS: Political Science and Politics 464, 464 (1995). But as before, the point is that there are other valid meanings of replication. Last time, my examples were based primarily on the writings of political scientists. This time, I'll provide examples from Lott’s own writings, which I found after posting my initial comments.

The first example comes from Lott’s article examining claims of racial discrimination at the polls in the 2000 election in Florida. For present purposes, the relevant material is in the appendix, which is labeled, “Using the Majority Report’s County-Level Data.” What he is referring to is the U.S. Commission on Civil Rights’ report, Voting Irregularities in Florida During the 2000 Presidential Election (2001). It’s a “majority” report because two members of the commission dissented. Here is what Lott says about the report:

Since the commission’s report, which was presented to the Senate, shows exactly what regression specifications were examined, I tried different specifications to replicate the commission’s results. The results (available from the author) showed that the simple cross-sectional county-level data were very sensitive to the specification used and that most specifications failed to show any statistically significant positive relationship between the percentage of voters who were African-American and the non-voted ballot rate. Thus, when the commission’s very own set of control variables is used, there is thus no real confidence that there is a positive relationship between the share of African-American voters and the ballot spoilage rate.

John R. Lott, Jr., “Nonvoted Ballots and Discrimination in Florida,” 32 Journal of Legal Studies 181, 211 (2003) (emphasis added). Since Lott used the same data, id. at 182, but analyzed them in different ways than the original authors, this reference to replication matches Sniderman’s second definition of the term, which is “the use of the same data, but not the same methods of measurement or estimation, to confirm the adequacy of the interpretation [and] reported results.”

The second example is from Lott’s point-by-point response to a critical column by syndicated columnist Michelle Malkin. In it, he references replication twice.

I spent years replacing the data lost in the hard disk crash. The county level crime data was replaced and given out to academics at dozens of universities so that they could replicate every single regression in More Guns, Less Crime. I have also made the data for my other book The Bias Against Guns available at http://www.johnlott.org/cgi-bin/login.cgi . The data for my other reserach [sic] has also been made available. The survey was also replicated and obtained similar results to the first survey and the new data has been made available since the beginning of the year. 

http://johnrlott.tripod.com/malkinsoped.html (emphasis added but note that the original version is entirely in italics). The first reference to replication in this paragraph matches Sniderman’s first definition, the verification of reported results. The second reference matches Sniderman’s third definition, “the use of a different data set and comparable measurement and estimation procedures, to validate the robustness of both the results initially observed and the interpretation originally given to them.”

Replication is a convenient term for describing several related activities. Even Lott’s writings demonstrate this.

Comments

[William Nuesslein]

In the vote count example noted in your blog, Dr. Lott did not say that the comission work could not be replicated, perse, rather he said that the results were not robust as that minor changes in input produed different results.

If Lott's results are not robust, then say so. One should not imply that the man made up the data or messed up in the analysis.

Mr. Levitt recently made some unfortunate remarks about the Stewart case on the Al Franken Show. It was clear that Ms. Stewart committed no crime with respect to her selling of ImClone stock, and that was not alledged in her trial. And Ms. Stewart was innocent of the charges actually made. Ms. Seymore invited the jury to believe that cheating was done. The jury accepted that invitation and convicted Ms. Stewart on the basis that she and her broker were covering up her cheating. Mr. Levitt and Mr. Franken both asserted that Ms. Steawrt cheated. According to them, Ms. Stewart just couldn't help doing such a dumb thing as cheating on the ImClone sale.

I love all three of Mr. Fraken, Mr. Levitt and Ms. Stewart. I wish that Mr. Fraken and Mr. Levitt would be more charitable and judicious in their comments.

[William Henderson]

Great post. bh.