At Wave 8 Understanding Society adopted a mixed-mode design, where 40% of participants were invited to complete online, a further 40% were initially approached for a face-to-face interview but then given the opportunity to complete online if they had not completed the face-to-face interview. The remaining 20% were only approached for a face-to-face interview.
The Covid pandemic accelerated the move to an online survey and although the Study has moved back to some face-to-face interviews, the majority of participants now complete their interview online. The mode of interview is known to affect survey data, as participants may answer questions differently depending on the mode of interview. For data users, the main survey user guide provides analysis advice for mixed mode data.
An experiment was fielded during the first year of Wave 8 data collection to assess the impact of the change in mode. The experiment involved a sequential design that permits the identification of mode effects in the presence of non-ignorable, non-random mode selection. This research describes a more general methodology based on novel structural moment models that characterizes the overall effect of mode on a survey by its effects on the moments of the survey variables’ joint distribution. The estimation procedure is adapted to account for nonresponse and complex sampling designs
Read the research: Estimating mode effects from a sequential mixed-mode experiment using structural moment models
Longitudinal datasets, like Understanding Society, are a key data collection tool used to estimate social change. There are concerns that mixing survey modes may affect to coefficients of change at the individual level. Research using Understanding Society has examined this, comparing single-mode (face-to-face) design versus sequential mixed-mode (web–face-to-face) design, and web versus face to face. Across 41 variables, this analysis found no differences in estimates of individual-level change across modes. However, correlations between intercepts and slopes, an estimate of convergence of respondents, were significantly different for most variables, which led to some biases in estimates of change.
Read the research: The impact of mixing survey modes on estimates of change: a quasi-experimental study
The COVID-19 pandemic meant that face-to-face interviewing had to be suspended on Understanding Society in March 2020. When face-to-face interviewing was suspended, all adult sample members were invited to complete their annual interview online, with a telephone follow-up for non-responders. The changes to the survey context and modes have the potential to affect the representativeness of respondents. Overall, the response rate for the 2020 sample was just 1.5 percentage points lower than the response in 2019, however this differs by sample characteristics. There are small but significant differences in the responding sample in 2020.
Read the research: COVID-19 and mode selection effects in Understanding Society
User Support has developed a self-paced online course for data users providing guidance calculating the effect of mixed modes using Understanding Society data. It is suitable for users of Understanding Society who are already familiar with the data structure of the Study and have knowledge of running statistical analysis using Stata. Users will learn about the mixed mode design in Understanding Society and how to assess the impact of the move to web on estimates based on the data (mode effects) as well as the challenges and opportunities a mixed methods approach brings to a survey.
