Authors
Summary
EXECUTIVE SUMMARY In recent years, survey agencies have increasingly employed a strategy of requiring interviewers to record a variety of different observations about all cases in their issued workload prior to first contact with the household. Because these observations are available for both responding and nonresponding households they are, in theory, potentially useful for the development of weighting schemes. However, little is known about the utility of these variables for increasing the accuracy of survey estimates through non-response weighting. In this report we identified all interviewer observation variables that have been included on major UK surveys in recent years. These can be broadly categorised as relating to characteristics of the area, of the household, and of the respondent. Existing studies on interviewer observation variables show that they are not very effective in reducing non-response bias. This is because they tend to be weakly related to response propensity and even more weakly related to key survey outcomes. Additionally, they appear to suffer from problems relating to measurement validity. An analysis of the innovation panel of the Understanding Society survey confirmed this general pattern. Only four variables from an extensive list of observations were found to be predictive of both response propensity and key survey outcomes. The power of even these variables in predicting survey outcomes was, however, weak. Analysis of the ONS Census link study showed that interviewer observation variables were not effective in improving the accuracy of survey estimates via non-response weighting. Many estimates exhibited an increase in bias after weighting and, on average, mean squared error was somewhat higher for the weighted than for the unweighted estimates. These findings suggest that interviewer observation variables should not be included on the National Survey for Wales. However, our recommendation is that observation variables should be included on the grounds that: there is no apparent cost saving from omitting them, it is possible that they may be of use in correcting for nonresponse bias in future rounds of the survey, and they have the potential to be of value for substantive as well as methodological analysis. The specific variables we recommend for inclusion are those that proved to be jointly predictive of response propensity and survey outcomes in our analysis of the Understanding Society Innovation Panel. Additionally, we recommend innovation in the sorts of measures that are collected. While the measures that have been developed to date are primarily oriented toward predicting response propensity, new measures which are intended to be more strongly correlated with key survey outcomes, might be more effective in improving survey accuracy through weighting. Some tentative suggestions are made for the sorts of measures that might be included.