An assessment of the potential utility of interviewer observation variables for reducing non‐response error in the National Survey for Wales: a report prepared for the Welsh Government by Patrick Sturgis and Ian Brunton-Smith

Publication type

Parliamentary Paper

Authors

Patrick Sturgis and Ian Brunton-Smith

Publication date

Summary


EXECUTIVE SUMMARY
 
In recent years, survey agencies have increasingly employed a strategy of
requiring interviewers to record a variety of different observations about all
cases in their issued workload prior to first contact with the household.
Because these observations are available for both responding and nonresponding
households they are, in theory, potentially useful for the
development of weighting schemes. However, little is known about the utility
of these variables for increasing the accuracy of survey estimates through
non-response weighting.
In this report we identified all interviewer observation variables that have been
included on major UK surveys in recent years. These can be broadly
categorised as relating to characteristics of the area, of the household, and of
the respondent.
Existing studies on interviewer observation variables show that they are not
very effective in reducing non-response bias. This is because they tend to be
weakly related to response propensity and even more weakly related to key
survey outcomes. Additionally, they appear to suffer from problems relating to
measurement validity.
An analysis of the innovation panel of the Understanding Society survey
confirmed this general pattern. Only four variables from an extensive list of
observations were found to be predictive of both response propensity and key
survey outcomes. The power of even these variables in predicting survey
outcomes was, however, weak.
Analysis of the ONS Census link study showed that interviewer observation
variables were not effective in improving the accuracy of survey estimates via
non-response weighting. Many estimates exhibited an increase in bias after
weighting and, on average, mean squared error was somewhat higher for the
weighted than for the unweighted estimates.
These findings suggest that interviewer observation variables should not be
included on the National Survey for Wales. However, our recommendation is
that observation variables should be included on the grounds that: there is no
apparent cost saving from omitting them, it is possible that they may be of use
in correcting for nonresponse bias in future rounds of the survey, and they
have the potential to be of value for substantive as well as methodological
analysis. The specific variables we recommend for inclusion are those that
proved to be jointly predictive of response propensity and survey outcomes in
our analysis of the Understanding Society Innovation Panel.
Additionally, we recommend innovation in the sorts of measures that are
collected. While the measures that have been developed to date are primarily
oriented toward predicting response propensity, new measures which are
intended to be more strongly correlated with key survey outcomes, might be
more effective in improving survey accuracy through weighting. Some
tentative suggestions are made for the sorts of measures that might be
included.

Links