Mechanical Turk & Sample Validity Problems

Many folks, especially those who conduct psychology experiments, will want to consider carefully Dan Kahan's (Yale) recent post about problems with relying on Mechanical Turk for data. More specifically, Kahan discusses "the invalidity of studies that use samples of Mechanical Turk workers to test hypotheses about cognition and political conflict over societal risks and other policy-relevant facts."

According to Dan, the "three decisive 'sample validity' problems" include: selection bias, prior, repeated exposure to study measures, and subjects' nationality misrepresentation. These problems, according to Dan, render Mechanical Turk samples particularly problematic for studies of culturally or ideologically grounded forms of “motivated reasoning.”