Separating systematic measurement error components using MTMM in longitudinal studies

This experiment extended multi-trait multi-method (MTMM) designs beyond allowing for separation of random and common method variance.  It identified three additional systematic variance components: acquiescence, social desirability, and extreme response. The research targeted opinions towards immigration with a repetition of six questions early in the questionnaire and late in the questionnaire, though of varying forms.

This experiment was carried in Waves 7, 8 and 9, with a fresh randomisation to treatment in each wave. Households within PSUs were randomly allocated to one of 56 different treatment groups which varied question wording, response options and the ordering in the questionnaire of these components. 

For each of the 6 items there are “positive” and “negative” formulations. “Positive” means that with a disagree-agree scale, the socially desirable direction will be towards the higher end of the scale.

Template for the questionnaire:

Table: Traits and social desirability direction for MTMM experiment

Trait number	SD direction	Item formulation
T1	Positive	The UK should allow more people of the same race or ethnic group as most British people to come and live here
T2	Positive	UK should allow more peopleof a different race or ethnic group from most British people to come and live here
T3	Positive	UK should allow more people from the poorer countries outside Europe to come and live here
T4	Positive	It is generally good for UK’seconomy that people come to live here from other countries
T5	Positive	UK’s cultural life is generallyenriched by people coming to live here from other countries
T6	Positive	UK is made a better place tolive by people coming to live here from other countries
T1	Negative	The UK should allow fewer people of the same race or ethnic group as most British people to come and live here
T2	Negative	UK should allow fewer peopleof a different race or ethnic group from most British people to come and live here
T3	Negative	UK should allow fewer people from the poorer countries outside Europe to come and live here
T4	Negative	It is generally bad for UK’seconomy that people come to live here from other countries
T5	Negative	UK’s cultural life is generallyundermined by people coming to live here from other countries
T6	Negative	UK is made a worse place tolive by people coming to live here from other countries

There are 8 different wordings of each item, corresponding to combinations of three factors: the higher- or lower-end being the socially desirable direction, the number of scale points, and whether agree-disagree or disagree-agree questions are used. These lead to 8 wordings W1-W8; an example formulation for trait one is given in the last column.

Table: Item wordings for MTMM experiment

Wording number	Social desirability	Number of scale points	Agree or Disagree	Required direction	Item formulation (using trait 1 as an example)
W1	Higher	2	AD	Negative	The UK should allow fewer people of the same race or ethnic group as most British people to come and live here
W2	Lower	2	AD	Positive	The UK should allow more people of the same race or ethnic group as most British people to come and live here
W3	Higher	11	AD	Negative	The UK should allow fewer people of the same race or ethnic group as most British people to come and live here
W4	Lower	11	AD	Positive	The UK should allow more people of the same race or ethnic group as most British people to come and live here
W5	Higher	2	DA	Positive	The UK should allow more people of the same race or ethnic group as most British people to come and live here
W6	Lower	2	DA	Negative	The UK should allow fewer people of the same race or ethnic group as most British people to come and live here
W7	Higher	11	DA	Positive	The UK should allow more people of the same race or ethnic group as most British people to come and live here
W8	Lower	11	DA	Negative	The UK should allow fewer people of the same race or ethnic group as most British people to come and live here

Instead of presenting each respondent with all 8 different wordings of the same items, any one respondent was asked only 2 different wordings of the items, one at the beginning and the second at the end of the questionnaire. There were (8,2) = 28 different questionnaire versions corresponding to the combinations of wordings. Moreover the ordering of the wordings was randomised, leading to 56 different conditions to be randomised in total.

Controlling variable on record hhsamp:

ff_mtmmw7 – Wave 7 treatment allocation: Takes the values 1-56 to allocate unique combinations of wording, response option and questionnaire ordering.  Coding is as in the table below.

ff_mtmmw8 – Wave 8 treatment allocation, coding as for Wave 7.

ff_mtmmw9 – Wave 9 treatment allocation, coding as for Wave 7

Table: Control variable values for MTMM experiment

	Randomised ordering 1 2			Randomised ordering 2 1
ff_mtmmw7 =	Question set early in q’re	Question set late in q’re	ff_mtmmw7 =	Question set early in q’re	Question set late in q’re
1	W1	W2	29	W2	W1
2	W1	W3	30	W3	W1
3	W1	W4	31	W4	W1
4	W1	W5	32	W5	W1
5	W1	W6	33	W6	W1
6	W1	W7	34	W7	W1
7	W1	W8	35	W8	W1
8	W2	W3	36	W3	W2
9	W2	W4	37	W4	W2
10	W2	W5	38	W5	W2
11	W2	W6	39	W6	W2
12	W2	W7	40	W7	W2
13	W2	W8	41	W8	W2
14	W3	W4	42	W4	W3
15	W3	W5	43	W5	W3
16	W3	W6	44	W6	W3
17	W3	W7	45	W7	W3
18	W3	W8	46	W8	W3
19	W4	W5	47	W5	W4
20	W4	W6	48	W6	W4
21	W4	W7	49	W7	W4
22	W4	W8	50	W8	W4
23	W5	W6	51	W6	W5
24	W5	W7	52	W7	W5
25	W5	W8	53	W8	W5
26	W6	W7	54	W7	W6
27	W6	W8	55	W8	W6
28	W7	W8	56	W8	W7

Variables affected on record indresp in Waves 7,8 and 9:

mtmm1q1e, mtmm1q2e, mtmm1q3e, mtmm1q4e, mtmm1q5e, mtmm1q6e, mtmm2q1e, mtmm2q2e, mtmm2q3e, mtmm2q4e, mtmm2q5e, mtmm2q6e, mtmm3q1e, mtmm3q2e, mtmm3q3e, mtmm3q4e, mtmm3q5e, mtmm3q6e, mtmm4q1e, mtmm4q2e, mtmm4q3e, mtmm4q4e, mtmm4q5e, mtmm4q6e, mtmm5q1e, mtmm5q2e, mtmm5q3e, mtmm5q4e, mtmm5q5e, mtmm5q6e, mtmm6q1e, mtmm6q2e, mtmm6q3e, mtmm6q4e, mtmm6q5e, mtmm6q6e, mtmm7q1e, mtmm7q2e, mtmm7q3e, mtmm7q4e, mtmm7q5e, mtmm7q6e, mtmm8q1e, mtmm8q2e, mtmm8q3e, mtmm8q4e, mtmm8q5e, mtmm8q6e, mtmm1q1l, mtmm1q2l, mtmm1q3l, mtmm1q4l, mtmm1q5l, mtmm1q6l, mtmm2q1l, mtmm2q2l, mtmm2q3l, mtmm2q4l, mtmm2q5l, mtmm2q6l, mtmm3q1l, mtmm3q2l, mtmm3q3l, mtmm3q4l, mtmm3q5l, mtmm3q6l, mtmm4q1l, mtmm4q2l, mtmm4q3l, mtmm4q4l, mtmm4q5l, mtmm4q6l, mtmm5q1l, mtmm5q2l, mtmm5q3l, mtmm5q4l, mtmm5q5l, mtmm5q6l, mtmm6q1l, , mtmm6q2l, mtmm6q3l, mtmm6q4l, mtmm6q5l, mtmm6q6l, mtmm7q1l, mtmm7q2l, mtmm7q3l, mtmm7q4l, mtmm7q5l, mtmm7q6l, mtmm8q1l, mtmm8q2l, mtmm8q3l, mtmm8q4l, mtmm8q5l, mtmm8q6l.

Variables affected on record indresp in Waves 7 and 8:

mtmmehh, mtmmemm, mtmmess, mtmmlhh, mtmmlmm, mtmmlss.

Variables affected on record indresp in Wave 8:

mtmml_temp, mtmme_temp.