Measuring how people’s social and economic circumstances change over time is a key purpose of household panel surveys. Levels of change are often overestimated in panel surveys. That is, responses to a question are often not consistent across interviews, even if the respondent’s situation has not changed. Various methods have been proposed to address these issues, but little is known about the mechanisms giving rise to the observation of spurious change. For this reason, a series of split-ballot experiments was incorporated into several waves of the Innovation Panel. The ultimate aim is to understand the processes that lead a respondent, whose situation has not changed from one interview to the next, to give a different response to a survey question in different interviews.
For the experiments, allocation was of households within PSUs. The variable that controls allocation to versions of Experiments 1-3 in Wave 2 is b_ff_changew2 on record b_indresp_ip:
Group 1 Version A of questions
Group 2 Version B of questions
Wave 2 experiment 1: Ambiguous wording. This contrasts questions where terms or definitions in the question were potentially ambiguous – contrasted with less ambiguous question wording. Respondents were randomly allocated to either version for the following questions:
Identifying disability status: items b_health_a and b_health_b on record b_indresp_ip
Identifying whether the respondent is in work: items b_jbhas_a1 through b_jbhas_b on record b_indresp_ip
Identifying whether the respondent saves regularly: item b_save_a1 through b_save_b on record b_indresp_ip
Wave 2 Experiment 2: Ambiguous instructions. This contrasts a question where the instructions were ambiguous in that response options were not mutually exclusive and there were no clear instructions about how to select the “main” category if more than one applied, with a “select all that apply” version of the same question whose instructions are less ambiguous:
Obtaining main labour market status: items b_jbstat_a through b_jbstat_d2 on record b_indresp_ip
Wave 2 experiment 3: Implicit or explicit questions about dates. This examines the effect of the clarity of instructions for questions about the dates of events. Questions in which the request for a date was implicit (e.g. “Since when have you…?”) were contrasted with questions where the request for the date was explicit (e.g. “In which month and year did you…?):
Obtaining dates of moves: items b_mvmnth_a through b_mvyr_b3, and b_plnowm_a through b_ plnowy4_b3, on record b_indresp_ip
obtaining dates of the onset of health conditions: items b_hconda_a through b_hconda_b3y on record b_indresp_ip
obtaining dates of joining private pension schemes: items b_ppyrs_a through b_ppyrs_b3y on record b_indresp_ip
Repetition of experiments 1-3 at wave 3
In Wave 3 the experiments were repeated with the same wordings, treatments and allocations. The controlling variable is c_ff_changew2 on record c_hhsamp_ip. Relevant variables are: c_health_a, c_health_b, c_jbhas_a1 to c_jbhas_b, c_save_a1, c_save_b, c_jbstat_a to c_jbstat_d2, c_mvmnth_a to c_mvyr_b3, c_plnowm_a to c_plnowy4_b3, c_hconda_a to c_hconda_b3y, c_ppyrs_a to c_ppyrs_b3y on record c_indresp_ip
Wave 3 experiment 4: Context and frequency. This experiment tests whether changes in the context of a question across waves can affect measures of change. The experiment uses a question about frequency of events, where the preceding question is either about high or low frequency events. The controlling variable is the IP2 variable allocating treatments to the measures of change experiments: c_ff_changew2 on record c_hhsamp_ip. Items about the frequency of political discussions follow either a high frequency item (c_mdafrq – media watching) or low frequency item (c_vtefrq frequency of voting). These items are on record c_indresp_ip. The political discussion items are c_poldisc1a through c_pldisc6a or c_poldisc1b through c_pldisc6b.
Repetition of experiments 1-4 at Wave 4
At Wave 4, Experiments 1 through 4 were repeated with identical wording and format. However, allocation to experimental group was rotated for some of the items in experiments 1, 3 and 4 (listed below), as compared to Wave 3. The controlling variable for the Wave 4 unchanged allocation is d_ff_changew2 on record d_hhsamp_ip. The controlling variable for the Wave 4 rotated allocation is d_ff_changew4 on record d_hhsamp_ip.
Table: Change in questions in wave 4 compared to waves 2 and 3
| Wave 4 | ||
| Wave 2 & Wave 3 | Version A | Version B |
| Version A | Same questions | Rotated questions |
| Version B | Rotated questions | Same questions |
Specifically, allocation at Wave 4 was unchanged (same questions) over prior waves for the following items: d_mvmnth_a to d_mvyr_b3, d_jbstat_a to d_jbstat_d2, d_plnowm_a to d_plnowy4_b3, d_jbhas_a1 to d_jbhas_b, d_save_a1, d_save_b on record d_indresp_ip.
Allocation at Wave 4 was rotated as compared to prior waves for the following items: d_mdafrq, d_vtefrq, d_health_a, d_health_b, d_hconda_a to d_hconda_b3y, d_ppyrs_a to d_ppyrs_b3y on record d_indresp_ip.
Note, the refreshment sample was allocated to all controlling variables for Experiments 1-4 and included in all treatments.
