The External Validity of College Student Subject Pools in Experimental Research: A Cross-Sample Comparison of Treatment Effect Heterogeneity

• Published in: Political analysis Vol. 27; no. 1; pp. 90 - 97
• Main Author: Lupton, Danielle L.
• Format: Journal Article
• Language: English
• Published: New York, USA Cambridge University Press 01.01.2019
• Subjects: Bargaining; College students; Crises; Demographics; Ethnicity; Experiments; Heterogeneity; International relations; Mintz, Steven; Political science; Politics; Reproducibility; Researchers; Respondents; Validity; generalizability; subject pools; treatment effects; student subjects; survey experiments
• ISSN: 1047-1987; 1476-4989
• DOI: 10.1017/pan.2018.42

Are student subject experiment pools comparable across institutions? Despite repeated concerns over the “college sophomore problem,” many experiment-based studies still rely on student subject pools due to their convenience and accessibility. In this paper, I investigate whether student subject pools are comparable across universities by examining how respondents across three student subject pools at distinct educational institutions perform on the same survey experiment about crisis bargaining between states. I argue that, due to selection biases inherent in university matriculation and the self-selection of students into experimental protocols, respondents across these subject pools will exhibit key demographic differences. I also examine whether respondents across these subject pools think similarly about international politics and respond comparably to experimental treatments. I find that, while there are significant demographic differences across subject pools, subjects across institutions respond similarly to experimental treatments—with the key exception of information regarding the regime type of a state. Furthermore, there is little evidence that these demographic differences impact conditional average treatment effects across subgroups. These findings carry critical implications for the use of student samples across political science and within international relations more specifically, particularly regarding the current replication crisis in the discipline.