An Introduction to Survey Experiments

Mark Pickup, Simon Fraser University


Survey experiments combine the power of causal inference from an experimental design with the external validity of representative surveys. Further, online survey experiments provide a highly flexible platform for conducting the experiment. This course will cover the theory, design and analysis of survey experiments. In terms of theory, the course covers topics such as the potential outcomes model of causal inference, and the logic of experiments. In terms of design, the course covers topics such as: types of randomization; blocking; priming and framing experiments; list and conjoint experiments; and incentivized experiments. As for analysis, the course covers difference of means (and proportions) tests, difference-in-differences tests, rank statistics, and OLS regression and randomized inference for testing treatment effects. In addition to a discussion of each of these topics, the course will include instruction on how to set up an online survey experiment and analyse the data.


This course runs January 27-31,2020.



Day 1: Review and Background

During the first day, we will review some of the basics of conducting surveys, with a focus on conducting web-based surveys. The lecture will also provide an overview of the potential outcomes model of causal inference, which will allow us to discuss the logic of experiments, and survey experiments in particular. This will include a discussion of different types of treatments in survey experiments and forms of randomization. The lecture and lab will also provide a review of difference of means/proportions tests and simple OLS regression, and discuss methods of analysis that may be new to students, including difference-in-differences tests, rank statistics, and randomized inference for testing treatment effects.

Day 2: List and Conjoint Experiments

Day 2 explores experiments designed to elicit preferences that are not easily obtained through traditional survey methods. These include sensitive opinions that are subject to social desirability effects, and preferences that require difficult trade-offs. In the lecture and lab, we will examine: 1) how the list experiment can be employed to measure the prevalence of sensitive opinions (e.g., anti-immigrant sentiment) in a population; and 2) how conjoint experiments can be used to measure the utility that respondents attached to difference preferences that may be in conflict with each other (e.g., characteristics of candidates in a democratic election).

Day 3: Priming and Framing Experiments

Day 3 examines experiments designed to test how context can change the attitudes and opinions one expresses. The lecture and lab will examine: 1) how framing experiments test how the informational environment can change a respondent’s expressed attitudes and opinions (e.g., how the provision of different types of arguments changes a respondent’s expressed support for a policy); and 2) how priming experiments test how the salience of different considerations can change a respondents expressed political choices (e.g., how making a political identity salient changes the respondent’s choices). In the lab component of the day we will also look at how to set up a very simple online survey experiment.

Day 4: Incentivized Experiments

A common concern in surveys is “cheap talk”. There is very little cost to a respondent if they give a response that feels good to them, even if it is not entirely true. For example, respondents may give responses that they think those conducting the survey want to hear. Alternatively, they may give responses that are consistent with their identity, even if it is not truly what they believe. In the case of partisan identity, the latter is known as partisan cheerleading. A potential solution to this is giving respondents an incentive to reveal their true beliefs and preferences. Monetary incentives are commonly paired with tasks in which the amount that the respondent receives depends upon the decisions she makes and potentially also the decisions of other respondents. This approach is more common in lab experiments but is increasingly being applied in survey experiments. In Day 4, we will cover the application of incentivized experiments within surveys.

Day 5: Cautions and Challenges

Although survey experiments can be very powerful, this requires the researcher to carefully connect the causal mechanism they wish to test to the survey design. Even then, questions of internal versus external validity, ethics, and response bias remain. In the last lecture, we will discuss potential problems when conducting survey experiments and the best ways to mitigate them.


Students are strongly recommended to take the Survey Research Design (offered in Module 1 of the the IPSA-USP 2020 Summer School and the Survey Analysis (offered in Module 2 the IPSA-USP 2020 Summer School) or the equivalent background in survey analyses.