Mixed mode experiments: web and face-to-face

At Waves 5, 6, 7, 8, 9, 10, 11, 15, 16 the Understanding Society IP was used to investigate the use of web interviewing. The incorporation of web into a mixed mode design has potential both to reduce survey costs and improve quality.

Please see the end of this section for a documentation of the various outcome variables for these waves.  

The Wave 5 sample had two components: the original sample, for which this was the 5th wave, and the refreshment sample, for which this was the 2nd wave. Households in both samples were randomly assigned within PSUs to one of two treatment groups. The controlling variable is w_ff_gridmodew5 on record w_hhsamp:

Group 1 – Face-to-face (one-third of each sample);

Group 2 – Mixed mode (two-thirds of each sample).

The distribution of the Wave 5 issued sample of households across samples and mode treatments is summarised in the table below. The randomisation was implemented across PSUs, so that each sampling point contained a mix of households in each treatment group.

The face-to-face treatment involved standard Understanding Society procedures. Each adult sample member (aged 16 or over) was sent an advance letter with an unconditional incentive, after which interviewers called to attempt computer-assisted personal interviewing (CAPI) interviews. The value of the incentive (in both samples) was subject to experimental allocation. In each household one person was asked to complete the household grid and household questionnaire. All household members aged 16 or over were asked for an individual interview and to complete a self-completion questionnaire, which was randomly allocated to be either a computer assisted self-interview (CASI) or a paper questionnaire booklet. Young people (in this group) aged 10-15 were administered a paper self-completion questionnaire.

Table: Number of households allocated to experimental groups

Note: Numbers shown are the numbers of households issued to the field, based on information held prior to the start of field work. During the course of field work, additional (split) households were identified. In the Refreshment sample, only responding households from their first wave (Wave 4 of the panel) were issued at Wave 5.

Fieldwork procedures for the web mode treatment were as follows: sample members aged 16 or over were sent a letter with the unconditional incentive, inviting them to take part by web. The letter included the URL and a unique user ID, which was to be entered on the welcome screen. A version of the letter was additionally sent by email to all sample members for whom we had an email address. For people who had indicated at previous waves that they do not use the internet regularly for personal use, the letter said that they would also have the opportunity to do the survey with an interviewer. Up to three email reminders were sent at 3-day intervals. Sample members who had not completed the web interview after two weeks were sent a reminder by post and interviewers started visiting them to carry out CAPI interviews. Note that this was the two-week web-only period in which households allocated to the additional £20 incentive conditional on whole household treatment at Wave 6 could qualify for the additional incentive (See Procedural Experiments: Incentives and Response). The web survey remained open throughout the fieldwork period.

The first household member to log on to do the web survey was asked to complete the household grid, which collects information on who is currently living in the household. The web grid included an additional question to identify who is responsible for paying bills. The household questionnaire could be completed by either this person or their spouse/partner. For these sample members the household questionnaire was displayed first, then leading on to the individual questionnaire. (The household questionnaire is relatively short – around 10 minutes – and collects household-level information such as housing tenure, rent/mortgage payments, expenditure, utility bills, household consumer durables and some measures of material deprivation.) Once one partner had completed the household questionnaire, it would not appear for the other partner.

At Wave 5, the youth survey was administered either on paper or by web, depending on the mode used by the parent(s). If the parent(s) had responded by web and we had their email address, an invitation was sent to the parent by email with a request to forward it to their child. If the parent had been interviewed in CAPI, the interviewer gave the youth the paper self-completion questionnaire. Otherwise, a questionnaire was sent by post.

The adult web questionnaire was based on the CAPI one, with some adaptations, e.g. incorporating interviewer instructions into question wording, removing references to showcards, and making “help” screens more respondent-appropriate. There were differences in the visual display of items between the web survey and the computer-assisted self-completion portion of the CAPI administered survey. Notably, self-completion components as part of CAPI were self-administered using the standard Blaise visual presentation ordinarily seen by interviewers. On the web survey, were a series of items utilized the same set of response options, the items were formatted in a grid rendering a difference in the visual presentation of these items across modes.

At Wave 5, the web survey was not suitable for completion using a small mobile device (e.g. smart phone). If a mobile device was used to access the log-on page, the respondent was automatically directed to a page requesting that they log on from a computer.

The Wave 5 mixed mode treatment also included two (crossed) experiments to test ways of increasing web response rates:

• Half the households were offered an additional conditional incentive: if all eligible household members completed the web survey within two weeks, they each received an additional £5. This was mentioned in the advance letters to all household members in this treatment group.
• Half the households were sent the advance letter and first email to arrive on a Friday. The other half were sent them to arrive on a Monday.

At Waves 6, 7, 8 and 9, households who were not completed at the end of the standard face-to-face fieldwork period, and were not adamant refusals, were contacted again in a ‘mop-up’ stage of fieldwork. This included non-responding individuals in partially responding households. The nature of the mop-up contact was differentiated, however, by mixed mode allocation:

• In the mixed mode group, the ‘mop-up’ contact was made by telephone. The telephone interviewer reminded the sample member that they could participate on the web, but was also able to administer the Wave 6 interview by telephone (CATI). Cases for which a telephone number was not known were not contacted again at the mop-up stage.
• The face-to-face group was contacted to offer a web interview during the ‘mop-up’ stage. Individuals were sent a letter with the URL of the web instrument and their unique log-on code. Those for whom we had email addresses, this invitation was sent by email. A few days later, a telephone interviewer contacted all those for whom phone number was known in order to remind them of the web questionnaire, and to administer a telephone interview if possible.
• At wave 7, The IP7 refreshment sample was not included in the web or telephone mop-up. Outstanding refreshment sample households and individuals continued to be attempted face-to-face during this period. At Wave 8 and 9, the IP7 refreshment sample was also included in the mop-up phase.

At Wave 8, allocation to mode remained broadly similar to past waves such that the IP6 and IP7 “Face-to-Face first” sample remained “Face-to-Face first” at Wave 8, and the IP6 and IP7 “web first” sample remained “web first”. However, a subgroup of households previously allocated to the ”web first” group were deemed to have very low web propensity and so moved to the “Face-to-Face first” group. Web propensity was determined through modelling observed characteristics, including mode of completion for previous waves. Web propensity has been calculated for the whole sample, including cases that at IP6 and IP7 were allocated to the “Face-to-Face first” group. The IP7 refreshment sample remained Face-to-Face first at IP8.

At Wave 9, 2/3 of households in the IP7 refreshment sample were randomly allocated to the “web first group” and the remainder to the “face-to-face first group”. Besides this change, the mixed mode allocation remained the same as IP8.

In all waves and subsamples, allocation to mode was made in advance of fieldwork at the household level and (in some cases) depended on prior mode allocation and Web mode response propensity as outlined above. The fed-forward variable ff_gridmodew8 controlled allocation to mode. Controlling variables are on record w_hhsamp_ip:

w_ff_gridmodew8

Group 1           F2F
Group 3           WEB

w_ff_lowwebw8 (Flag for low web-propensity)

-9         Inapplicable, previous wave F2F cases
0          Web allocation
1          Low Web propensity

At Waves 6-9, the mixed mode treatment included an incentive experiment to test an incentive plan that might increase whole household web response rates:

• One-third of mixed mode households were offered a £30 pound unconditional individual incentive to participate
• One-third of mixed mode households were offered a £10 unconditional individual incentive with an additional £20 per individual if the whole household completed within the two-week web-only period.

Please see Experiment 7.1 for details on these incentive experiments.

At Wave 6 to 9, the youth survey was administered only on paper. Interviewers distributed paper self-completion questionnaires to youth whose parent(s) were interviewed in CAPI. Youth whose parents were interviewed by web were sent a questionnaire by post.

At Wave 10, the allocation to mode remained broadly similar to past waves such that IP1/IP4/IP7 Face-to-Face sample remained Face-to-Face and the IP1/IP4/IP7 Mixed-Mode sample remained Mixed-Mode at IP9. The wave 10 refreshment sample was allocated to Face-to-Face. The ‘mop-up’ stage at the end of the standard fieldwork period was as in prior waves. The controlling variable randomised at the household level is:

ff_gridmodew10 in record J_HHSAMP_IP:

1          F2F
3          WEB

At Wave 11, the allocation to mode remained the same as in previous waves for the IP1/IP4/IP7 samples, with Face-to-Face allocated sample members remaining Face-to-Face at IP11, and the IP1/IP4/IP7 Mixed-Mode sample remaining Mixed-Mode at IP11.

As with all previous refreshment samples, the IP10 refreshment sample was allocated to Face-to-Face only in the wave in which it was recruited (IP10). For IP11, 1/3 of this sample was randomly allocated to the Face-to-Face only design, with the other 2/3 allocated the Mixed-Mode design.   

The IP11 refreshment sample was randomly allocated to modes from the start: ½ of households were allocated to Face-to-Face only and ½ to Mixed-Mode. The IP11 reserve refreshment sample that was issued towards the end of fieldwork was however allocated to Face-to-Face only. The contact letter for households allocated to the Mixed-Mode design included a URL and sign-in information to complete the survey. The variable controlling allocation for this experiment at IP11 is:

ff_gridmodew11 in record k_hhsamp_ip:

1          F2F
3          WEB

The variable that identifies whether the IP11 refreshment sample members were part of the initial batch or the reserve sample is ff_samplestatusw11 in record k_hhsamp_ip:

0          Existing household

1          Main refreshment

2          Reserve refreshment

The IP11 refreshment sample included an addition experiment with the invitation to the mixed mode survey (see 7.16 below).

At Wave 12, the goal was to compare standard design modes, i.e. CAPI and Web to a standard of nurse collected biomarkers. Three strands of data collection were therefore conducted: Nurse-led, Interviewer-led (standard CAPI), and mixed-mode (sequential web-CAPI). Both interviewer- and nurse-led data collections included measurements of blood pressure and height and weight. Nurse-led interviews included an additional collection of full blood, dried blood spots and a hair sample. Interviewer-led and web respondents were given a kit containing materials to enable dried blood spots and hair samples to be taken independently and returned at a later time (see Section 9.28). All participants were asked to collect a blood pressure measurement prior to completion of visit or web data collection, incorporating an experiment to examine prosocial versus informational content on response and quality of measurement (see Section 7.18).

One third of households were allocated to each of these three modes of data collection. Allocation to mode at IP12 was independent of any allocations at prior waves or sample status. As in past waves, respondents in the web-first condition were first invited to complete an interview on the web; those that did not complete the web survey were then assigned to an interviewer for completion.

The variable controlling allocation for this experiment at IP12 is ff_gridmodew12 in record l_hhsamp_ip:

1          Face-to-face
3          Web-first

4          Nurse

At Wave 13, respondents were allocated to one of two modes: Face-to-Face first and Web-first. However, due to the outbreak of COVID-19 and the measures put into place to limit its spread through personal contact, all households were invited to complete the survey in the web-first design. All follow-ups of web non-respondents by an interviewer were through telephone. Overall, the change in modes had little impact on content asked to respondents. The one exception was that those responding to the telephone follow-up were not asked most of the self-completion modules, which is standard practice in past years of telephone mop-up phases.

While not implemented in practice due to the outbreak, below are details of the IP13 mixed-mode experiment. Given allocation will likely carry over waves, as in the past, this design will likely reflect what will be done in IP14. The allocation to Face-to-Face first or Web-first followed what was done from IP5 through IP11, however, allocation was new, so respondents were not necessarily in the same mode condition as they were in past waves. As with these previous allocations, about 1/3 of the households were allocated to the Face-to-Face first design, with the other 2/3 allocated to the Web first design.

The variable controlling allocation for this experiment at IP13 is ff_gridmodew13 in record m_hhsamp_ip:

1          Face-to-face first
3          Web-first

At Wave 14, all sample members were issued to Web-first and followed up by telephone, due to the ongoing COVID-19 restrictions.

At Wave 15, the prior mixed mode allocations were resumed. The IP14 refreshment sample households were allocated 2/3 to web-first and 1/3 to face-to-face first.

The variable controlling allocation for this experiment at IP15 is ff_gridmodew15 in record o_hhsamp_ip:

1          Face-to-face first
3          Web-first

At Wave 16, the prior mixed mode allocations were again maintained. The variable controlling allocation for this experiment at IP16 is ff_gridmodew16 in record p_hhsamp_ip:

1          Face-to-face first
3          Web first

Indicators of Mode Allocations and Outcomes in Waves 5, 6, 7, 8, 9, 10, 11, 15, 16

The controlling variables in the data which control mode allocation and related variables containing information on interview outcomes are outlined below. The letter “w” indicates the wave, taking the letters “E”, “F”, etc.

On record w_hhsamp_ip:

• w_ff_gridmodew* contains the treatment allocation, where * corresponds to the wave (so e_ff_gridmodew5 is the wave 5 allocation variable)
• w_ivfhqo indicates whether household questionnaire was completed, w_hhgridmode indicates the mode in which the household grid was completed
• w_hhintmode indicates the mode in which the household questionnaire was completed
• w_ivfho provides a summary of which instruments were completed by the household
• w_hhmodes indicates whether the instruments a household completed were done by face-to-face, web or telephone only, or by using a mix of modes. Note, a cross-tab against w_ivfho can be used to identify whether the household completed all instruments or whether some are missing at a particular wave.
• w_web_outcome indicates the household outcome at end of web-only period (available for waves 5, 6, 7 only)
• w_f2f_outcome indicates the household outcome at end of face-to-face interviewing period (available for waves 5, 6, 7 only)
• f_tel_outcome indicates household outcome at end of telephone mop-up period (available for waves 5, 6, 7 only)
• w_ff_lowwebw8 flag for low web-propensity (from wave 8 onwards)

on record w_indall_ip

• w_ivfio indicates the individual interview outcome
• w_indmode indicates the mode in which the individual interview was completed