The way you ask subjective evaluation questions (e.g., satisfaction) may have a big influence on the types of answers that you get. A series of experiments with question wording, format and placement have been run in Waves 1-3 and 5 of the IP. Experiments across Waves 1-3 concerned the measurement properties of various questions on satisfaction whereas experimentation in Wave 5 concerned the nature of the judgement. In addition, an experiment in the Wave 5 youth questionnaire examines satisfaction measurement amongst young people.
In Wave 1, the experiment compared 11 and 7 point scales for job satisfaction. There is debate about the number of response categories which is substantively meaningful for respondents and for analysis.
Within PSUs, households were randomly allocated to one of two conditions. All interviewed adults within household received the same experimental treatments. The conditions are:
Group 1 11 point scale, no showcard, only end-points labelled
Group 2 7 point scale, no showcard, only end-points labelled
The variable that controls allocation to treatments is a_group2 on the record a_hhsamp_ip. The variables a_jbsat_g1 and a_jbsat_g2, for groups 1 and 2 respectively, on the record a_indresp_ip contain the substantive information.
At Wave 2, the experiment expanded to other satisfaction items about the participant’s satisfaction with their health, family income, leisure, job (if applicable) and their life overall.
As well as the mode and showcard experiments, households were independently assigned to treatment groups formed by varying question design, delivery and position within the interview.
All eligible adults within a household received the same experimental treatment. The Table below sets out the 10 treatment groups at Wave 2 for this experiment.
| Table: Definition of experimental groups in Wave 2 experiment: measurement of satisfaction. | ||
| Group | Response Mode | Timing of Question |
| 1 | CASI Full-labels | Late in questionnaire |
| 2 | CASI Polar-labels | Late in questionnaire |
| 3 | F2F + showcard Full-labels | Late in questionnaire |
| 4 | Tel + F2F Full labels: branched | Late in questionnaire |
| 5 | F2F + showcard Polar labels | Late in questionnaire |
| 6 | Tel + F2F Polar-labels | Late in questionnaire |
| 7 | F2F + showcard Full-labels | Early in questionnaire |
| 8 | Tel + F2F Full labels: branched | Early in questionnaire |
| 9 | F2F + showcard Polar-labels | Early in questionnaire |
| 10 | Tel + F2F Polar labels | Early in questionnaire |
| CASI = Computer assisted self interview; Tel = telephone; F2F = Face-to-face | ||
The standard questions have an initial question, “How dissatisfied or satisfied are you with the following aspects of your situation: (a) your health; (b) the income of your household; (c) the amount of leisure time you have”? This was followed by the question, “Using the same scale, how dissatisfied or satisfied are you with your life overall?” Respondents who were employed or self-employed were in addition asked, “All things considered, which number best describes how dissatisfied or satisfied you are with your job overall?”
The responses were measured using a seven-point scale. There were 3 different delivery methods (showcard, oral, CASI). Groups 3, 5, 7 and 9 had the response categories on a showcard. Groups 4, 6, 8 and 10 had no showcard; the question was purely oral. Groups 1 and 2 were presented with the computer and asked to complete the question by themselves (CASI).
The response scale was presented in three different ways. For groups 1, 3 and 7 each of the points on the seven-point scale was labeled (Full labels: 1-stage): 7 Completely satisfied; 6 Mostly satisfied; 5 Somewhat satisfied; 4 Neither satisfied nor dissatisfied; 3 Somewhat dissatisfied; 2 Mostly dissatisfied; 1 Completely dissatisfied. Participants in groups 4 and 8 were also able to answer using the fully-labelled scale, but the question was broken into two parts (branched), with the participant first being asked, “How dissatisfied or satisfied are you with your (life/job) overall? Would you say that you are… (1 Dissatisfied; 2 Neither dissatisfied nor satisfied; 3 Satisfied).” If the participant indicated that they were either dissatisfied or satisfied they were asked the follow-up question, “Are you somewhat, mostly or completely (dissatisfied/satisfied) with your (life/present job) overall? (1 Somewhat; 2 Mostly; 3 Completely).” The third treatment, for groups 2, 5, 6, 9 and 10, is the “Polar labels” option. In this treatment group, only the labels for the extreme points on the scale were conveyed; i.e., completely dissatisfied and completely satisfied.
The timing of the job satisfaction question in the questionnaire was fixed for all participants who had a job, following a section about employment or self-employment. The life satisfaction questions were either asked early in the interview (about a quarter of the way through, Groups 7-10) or late in the interview (very near the end, Groups 1-6).
The controlling variable for the job satisfaction split-ballot design is b_ff_jobsatw2 on the record b_hhsamp_ip. Values of this variable correspond to groups 1-6 in the table above.
The controlling variable for the split-ballot design of the remaining satisfaction items is b_ff_lifesatw2 on the record b_hhsamp_ip with values corresponding to the entire set of experimental treatments outlined in the table above.
The substantive data for job satisfaction can be found in the items b_jbsat_a through b_jbsat_f on the record b_indresp_ip. The substantive data for the remaining satisfaction items can be found b_lfsat1_a through b_lfsato_j on the record b_indresp_ip.
At Wave 3 and Wave 6 this experiment was repeated with a rotation in experimental treatments. The life and job satisfaction split-ballot question wording experiment carried at Wave 2 was repeated at Waves 3 and 6. The experimental allocation at Waves 3 and 6 varied the Wave 2 allocation, however. As well as the showcard experiments, described above, households were independently assigned to experimental groups formed by varying question design, delivery and position within the interview. All eligible adults within a household received the same experimental treatment. Also, while the Wave 2 allocation was nested within the showcard experiment, the Waves 3 and 6 allocations were not. Therefore, a separate showcard was required for the satisfaction items for interviewers not otherwise issued with showcards at Wave 3, and there was no showcard experiment in Wave 6. There was no manipulation of placement for the life satisfaction items.
This manipulation was of households within PSU’s, therefore interviewers received different versions of these questions depending on which household they were interviewing.
Note that there was an error in the implementation of the Wave 3 satisfaction experiment, which meant that some respondents were asked the life-satisfaction questions twice, once at the beginning and once at the end of the questionnaire. For details see section Known Data Issues. For this reason, the exact experimental allocation from Wave 3 was replicated in Wave 6.
The questions at Waves 3 and 6 match Wave 2. The standard question-set involved an initial question, “How dissatisfied or satisfied are you with the following aspects of your situation: (a) your health; (b) the income of your household; (c) the amount of leisure time you have”. This was then followed by the question, “Using the same scale, how dissatisfied or satisfied are you with your life overall?” Those participants who were employed or self-employed were asked, “All things considered, which number best describes how dissatisfied or satisfied you are with your job overall?” The responses were measured using a seven-point scale. Whereas in Wave 2, 10 different experimental treatment groups captured variation in the presentation of satisfaction items, only 6 different groups were used at Waves 3 and 6. The group allocation was permuted for Waves 3 and 6 to achieve higher sample sizes across groups when waves are pooled.
For the Life Satisfaction Items, the permutation of treatments is as follows:
| Table: Treatment groups in Wave 2 and Waves 3 and 6 for experiment on measurement of life Satisfaction | |||
| Group | Treatment at Wave 2 | Group | Treatment at Waves 3 and 6 |
| 1 | Full labels, CASI, beginning | 2 | Polar labels, CASI |
| 2 | Polar labels, CASI, beginning | 5 | Polar labels, showcards |
| 3 | Full labels, showcards, Beginning | 1 | Full labels, CASI |
| 4 | Full labels, unfolding design, Beginning | 6 | Polar labels, no showcards |
| 5 | Polar labels, showcards, Beginning | 3 | Full labels, showcards |
| 6 | Polar labels, no showcards, Beginning | 4 | Full labels, no showcard, branched rating |
| 7 | Full labels, showcards, end | 4 | Full labels, no showcard, branched rating |
| 8 | Full labels, unfolding design, end | 3 | Full labels, showcards |
| 9 | Polar labels, showcards, end | 6 | Polar labels, no showcards |
| 10 | Polar labels, no showcards, end | 5 | Polar labels, showcards |
At Wave 3, the controlling variable is c_ff_lifesatw3 on record c_hhsamp_ip while at wave 6 the controlling variable is f_ff_lifesatw3 on record c_hhsamp_ip. In both Wave 3 and Wave 6, this variable is coded:
1 fully labelled CASI, end of interview
2 polar point labelled CASI, end of interview
3 fully labelled with showcards, end of interview
4 fully labelled without showcards, unfolding design, end of interview
5 polar-point labelled with showcards, end of interview
6 polar-point labelled without showcards, end of interview
Groups 3 and 5 had the response categories presented visually on a showcard. Groups 4 and 6 had no visual cue, the question was purely oral. Groups 1 and 2 were presented with the computer and asked to complete the question by themselves (CASI).
As well as the different delivery methods at Waves 3 and 6 (showcard, oral, CASI), the response scale was presented in three different ways. For groups 1, 3 and 4 each of the points on the seven-point scale was labelled; 7 Completely satisfied; 6 Mostly satisfied; 5 Somewhat satisfied; 4 Neither satisfied nor dissatisfied; 3 Somewhat dissatisfied; 2 Mostly dissatisfied; 1Completely dissatisfied. This is the “Full labels: 1-stage” response scale. Participants in group 4 were also able to answer using the fully-labelled scale, but the question was broken into two parts, with the participant first being asked, “How dissatisfied or satisfied are you with your (life/job) overall? Would you say that you are… (1 Dissatisfied; 2 Neither dissatisfied not satisfied; 3 Satisfied).” If the participant indicated that they were either dissatisfied or satisfied they were asked the follow-up question, “Are you somewhat, mostly or completely (dissatisfied/satisfied) with your (life/present job) overall? (1 Somewhat; 2 Mostly; 3 Completely)”. The third treatment, for groups 2, 5, and 6 is the “Polar labels” option. In this treatment group, only the labels for the extreme points on the scale were conveyed; i.e., completely dissatisfied and completely satisfied.
Wave 3 substantive data for job satisfaction can be found in the items c_jbsat_a through c_jbsat_f on the record c_indresp_ip. Comparable wave 6 variables are f_jbsat_a through f_jbsat_f on the record f_indresp_ip.
Wave 3 substantive data for the remaining satisfaction items can be found c_lfsat1_a through c_lfsato_f on the record c_indresp_ip. Comparable variables for wave 6 are f_lfsat1_a though f_lfsato_f on the record f_indresp_ip.
