What is panel conditioning and what can we do about it?
Professor Peter Lynn
If people take part in our survey over many years, do they start giving different answers compared to when they took part the first time? Is it a problem? And what can we do about it?
Panel conditioning is a phenomenon that can affect the quality of longitudinal survey data. It’s something people involved in surveys have been aware of for some time.
Back in 2009, I described it in my book Methodology of Longitudinal Surveys: “Panel conditioning refers to the possibility that survey responses given by a person who has already taken part in the survey previously may differ from the responses that the person would have given if they were taking part for the first time. In other words, the response may be conditioned by the previous experience of taking part in the survey.”
What do we know?
Panel conditioning can change someone’s behaviour. For example, if you ask a mother whether her children have been immunised, she may think, “Oh, I need to do that, don’t I?” Then, when you interview her again, the children have all been immunised, so the answer has changed. That’s good for them, but arguably not for the data, if they’re no longer representative of the population as a whole.
Panel conditioning can also influence a person’s reporting behaviour. This tends to be commonest when the interviewing is relatively frequent, so respondents can remember fairly well what happened in the previous interview. Some respondents might then give different answers in order to get the interview over quicker. This can happen particularly when there are a lot of follow-up questions.
Understanding Society, though, is insulated from these effects to some extent because the nature of our survey is different. We don’t have many questions which would make people actively think about changing their behaviour – and we have a one-year gap between interviews. People are unlikely to remember what they said, or the questions they were asked, in any great detail. They probably only have a general memory of what the topics were.
What’s most likely for us?
The form of panel conditioning that may be most likely with Understanding Society is that respondents learn to trust the interviewer. Paul Fisher, a research fellow at the Institute for Social and Economic Research (the home of Understanding Society), has written a paper which shows that people’s reporting of their income seems to get better from the second time they are interviewed.
That’s the kind of subject on which people are wary of giving accurate answers – none of us likes discussing our finances with a stranger. So, this is an example of positive panel conditioning: we’re getting a better quality of data than we would if we hadn’t repeated the interviews.
Certain types of question carry a greater risk of panel conditioning than others. For example, questions which open up a ‘loop’ of follow-up questions. We sometimes ask about a respondent’s three closest friends. Then we ask a series of questions about each of the friends – how many of them live nearby, how similar they are in age, income and ethnic group, for example, whether they’re also a family member, and so on.
We may also ask if the children in the family have some kind of day care arrangement. If the answer’s ‘yes’, we then ask what is it, who provides it, how often, etc.
With these, there is the potential for people to learn that if they answer the question in a certain way, they won’t get the follow-up questions, so the interview will be over quicker. That is something to be concerned about, but it is less likely to be a big problem on Understanding Society, for three reasons:
- First, because our interviews are annual, the effect on respondents is much less than it would be if we were asking the same questions more frequently, for example every month.
- Second, the questions we ask are serious ones, looking for factual information. People are unlikely to say, “No my child doesn’t go to nursery” just to avoid answering questions about them. Whereas, in a different type of survey, you might be more likely to say, “No, I haven’t travelled on a Virgin train” if you know you’re going to get a whole load of questions about that because you’ve just answered a whole load of follow-up questions about, let’s say, Greater Anglia trains.
- Third, a study authored by Understanding Society’s Associate Director of Innovations, Annette Jäckle, showed that the effect of looping questions depends how the questions are structured. So, if I ask you ten questions about your experience of travelling on Greater Anglia, and then: “Have you travelled on a Virgin train?”, you can immediately spot that I’m about to ask 10 follow-up questions about another train company – creating a greater risk of people saying no. But if I say, “Have you been on a Greater Anglia train?”, then “Have you been on a Virgin train?” and so on – and then ask the follow-up questions later on in the interview, your response to the initial questions cannot be influenced by the presence of the follow-up questions because you are not yet aware of them. Where questions lead to follow-ups, we’ve tried to design them so that it’s not so obvious that the follow-up is a result of having answered the initial question. That makes it less likely that respondents will try to take a shortcut through the interview.
Panel conditioning is more likely to happen with attitude or opinion questions about something on which the respondent doesn’t have strong views. Then, their views are more malleable, and they’re more susceptible to being influenced by the fact that you’ve raised the topic and made them think about it. But, because our questions are mainly factual, about things happening in our respondents’ everyday lives, they’re less susceptible to this kind of conditioning.
We think our respondents can see that Understanding Society is important, so – although it takes a little while – it’s obvious why we need to ask all those questions. We’re not asking them a lot of similar questions about different products. We move from topic to topic, and because they see the need for questions on these important topics, that reduces the risk of people trying to take a shortcut.
We are very aware of panel conditioning, but the broad design and the topics we cover mean that Understanding Society is less susceptible to it than many other surveys. And we rotate questions and design the survey in a way that minimises the risk. We’re confident, then, that our data is robust and representative of the UK population – and researchers can continue to rely on that.
Peter is Understanding Society Associate Director for Methodology and Professor of Survey Methodology at the University of Essex