Analysis advice for mixed mode data

Using different modes during a survey can affect how respondents’ answer the same questionnaire.

Despite this possibility, the convenience and potential cost savings, especially relative to face-to-face or in-person interview mode (referred to as CAPI, Computer Assisted In-person Interviews) have led Understanding Society to adopt a push-to-web mixed-mode design, starting at Wave 8.

The presence of mode effects means that, potentially, those who completed the questionnaire using web may provide different answers online than they would have done had they been interviewed face-to-face (CAPI) or by telephone (CATI, Computer Assisted Telephone Interviews). Given that 29% of Wave 8 individual interviews were carried out online (i.e., self-completion on the web, referred to as CAWI, Computer Assisted Web Interviews), this means that the introduction of a mixed modes design could affect longitudinal analyses involving data from Wave 8 and earlier, predominantly CAPI, waves.

To estimate the impact of this change, an experiment was carried out during the first year of Wave 8 fieldwork (details provided below). Furthermore, at the same time, a random subsample of households was ring-fenced in order that the survey could be run in exactly the same way as it had been in earlier waves. This ring-fenced sample was set up to be a benchmark against which the data from the new push-to-web design could be compared. The experiment was neither run during the second year of Wave 8 fieldwork nor in any of the subsequent waves. From onwards Wave 8 Year 2 until the start of the Covid-19 pandemic in March 2020, the mode allocation was as follows:

• 20% were ring-fenced (but those who wanted to shift to web were allowed)
• Among the rest, using the IP model, an increasing % of those who were most likely to respond by web were allocated to web-first. These percentages were 40% in Wave 8 Year 2, 60% in Wave 9, 70% in Wave 10, 80% in Wave 11 (identified by w_issue_mode = 3 in w_hhsamp).

After the pandemic lock-down began, all households were issued web-first with a telephone follow-up. When face-to-face interviewing resumed in April 2022, the ring-fenced sample was removed as its usefulness was lost (given that they had not been interviewed in-person for two waves). Around 12% of households who were less likely to participate by web were allocated to CAPI-first and rest to CAWI-first.

Mode effect Vs Measurement error

It is important to recognise that a substantively significant difference between the answers obtained under CAWI and under CAPI or CATI does not automatically imply that the web answer is ‘worse’. A mode effect of CAWI is not the same as measurement error unless one is prepared to argue that the survey instruments under CAPI is a Gold Standard with negligible measurement error. It is more accurate to say that CAPI is a benchmark for comparison with CAWI data. D’Ardenne, Collins et al. (2017) discuss theories of response (fear of disclosure, social desirability bias, positivity bias, satisficing) and how mode effects could vary from question to question depending on features of the question. Generally, which mode is ‘best’ depends on the nature of the question and the way it is asked, and is difficult to determine.

Wave 8 mode experiment

The data from the experiment allow the estimation of the effect of web mode on the data collected for Understanding Society.

As noted above, at Wave 8 Year 1, the proportion of the web-first sample was increased: adults in around 40% of households were invited to participate online, with non-respondents followed up in-person. This is referred to as CAWI-first or web-first. This group included both previous-wave participants and non-participants. Adults in the remaining 60% of households were issued, as previously, directly to CAPI-first.

The allocation of households to the web-first mode was not done randomly. Aside from 20% of the sample that were ring-fenced as face-to-face, the remaining sample was allocated using a model created using the Understanding Society Innovation Panel (IP). The IP had been mixed-mode, with randomly allocated web-first and CAPI-first samples, since the fifth wave (IP5). The IP data were used to estimate a model which aimed to maximise the probability of a household fully responding online whilst minimising the chance that non-responders to the web invite would then refuse to participate face-to-face. The model built using the IP was then applied to the main-stage sample to generate a web propensity score. Those households with the lowest propensity to respond by web were issued to face-to-face interviewers (16%). The remaining sample was randomly allocated between web-first (40%) and CAPI-first (24%).

Adjusting for mode effects

Unfortunately, it is not possible to devise a fix to adjust the results of a mixed-mode analysis to remove the mode effects. The best that can be done is to predict what the results would have looked like had another mode been used instead. For example, the difference between the observed data from Understanding Society using push-to-web and what would have been observed had CAPI been used.

Clarke and Bao (2022) developed statistical methods to estimate the effect of mode on the survey variables from mixed-mode survey data, focussing on the experimental data like that from the first year of Wave 8 fieldwork but also (under stronger assumptions) from non-experimental mixed-modes data.

They estimated the mode effects for a range of survey variables but found little evidence for mode effects that would affect the results of a multivariate or multivariable analysis, so we conclude that, generally, mode effects are not a major issue for Understanding Society. We take these effects to be hold in subsequent waves of the main survey where a similar push-to-web design was used.

However, the enormous range of variables measured by Understanding Society means that not every variable was assessed, and we appreciate that users may be concerned about the presence of mode effects for particular variables.

Users can complete a training course showing how to use the methods we have developed to estimate the effect of mode on the measurement of their survey variables: Guidance on calculating the effect of mixed modes using Understanding Society.

We also continue to offer the following advice for those users who wish to investigate the impact of web without using the methods described above:

1. Do not use the ‘indicator method’ for a regression/multivariable analysis: The indicator method is simply to include a dummy variable that indicates whether the user chose web or CAPI as predictor variable in the regression analysis. However, despite its popularity, it was found that this approach is generally ineffective because it can often lead to badly biased results.
2. For data from Waves 8-11 (until start of the Covid-19 pandemic), a simple sensitivity analysis can be carried out to compare the estimates obtained using only the ring-fenced sample with those obtained using the remaining data: The ring-fenced sample is a random sample of 20% of households for which the survey aimed to administer CAPI-only, as in previous waves. The variable h_ringfence identifies members of this sample. (Users should note where CAPI interviews were not obtained, respondents were offered the option of completing online instead; a crosstab of the variables w_indmode and w_issue_mode identifies the web and telephone cases to exclude from the CAPI-only sample.) To test whether the results of a regression analysis are different in the ring-fenced sample from those in the mixed modes sample, the analyst can 1) include h_ringfence as a main effect in the model, and 2) include the interactions between h_ringfence and each predictor variable in the model. We recommend that the survey design and weights are accounted for when performing this analysis. If any of the interactions created in step 2 are statistically significant, this indicates the potential presence of mode effects. If the results are significant and you are unsure of how to proceed, it is recommended that you consult a statistician on your team to discuss.