"Is that still the same? Has that changed?"
How question wording can help surveys more accurately measure change and stability.
One of the primary aims of household panel studies, like Understanding Society, is to measure change over time. New jobs, changes in income, new relationships, health issues, moving house – household panel studies aim to accurately record the life changes that our participants go through.
When measuring change, researchers interpret changes in answers over time as changes in participants’ situations. However, when respondents give a different answer in one interview from the next, this can also be due to reporting errors. As a result, panel surveys often overestimate change. To help reduce this measurement error, and to help survey participants recall information about changes at their annual interview, household panel studies use dependent interviewing, where the answers from the previous year are pre-loaded in the survey and participants just have to say whether the information is still accurate. This technique makes it quicker and easier for the participant, but can sometimes lead to errors in responses as participants may choose to answer in the quickest way, “yes, that’s the same”, rather than the most accurate way.
Researchers Annette Jäckle, from Understanding Society, and Stephanie Eckman, at the Institute for Employment Research in Germany for the period of the research and now at RTI International, wanted to know whether the phrasing of dependent interviewing questions makes a difference to how accurately participants answer. In Understanding Society participants are asked “is that still the same?” and “has that changed?”. Which question used tends to be based on what sounds the most natural in the conversation flow of the interview, but does one question form lead to more accurate answers?
The Understanding Society Innovation Panel and the German Measuring Change survey were used to test the two forms of question. Jäckle and Eckman allocated different questions to different participants. Some were asked “is this still the same?”, some were asked “has that changed?” and some were asked the hybrid “is this still the case or has it changed?” or “has that changed or this still the case?”. The Measuring Change survey is linked to administrative data, allowing the researchers to check on the accuracy of survey answers.
Does the question make a difference?
Yes, it does. Using “has this changed?” led to more participants reporting changes, but these changes were not always accurate. The question seemed to encourage participants to report changes, but these were then reversed in later questions in the survey. In both studies using the “still” questions (“is this still the same?”, “is this still the case or has it changed?”, “has that changed or is this still the case?”) produced similar results, with lower rates of reported change than the “has that changed?” question, but more accuracy in participants answers.
“Taking together the evidence from this research, as well as findings from previous research, we believe that the “still” format provides the best data quality and recommend that panel surveys use this wording. It reduces the over-reporting of change seen with the independent question.” Annette Jäckle and Stephanie Eckman
You can read the full research paper here: https://academic.oup.com/jssam/advance-article/doi/10.1093/jssam/smz021/5532310