The accuracy of using paradata to detect and understand interviewer question-reading deviations

Publication type

Conference Paper

Author

Jennifer Kelley

Publication date

Summary

Deviations from reading survey questions exactly as worded may change the validity of the questions, thus increasing measurement error. Hence, organizations train their interviewers to read questions verbatim. To ensure interviewers are reading questions verbatim, organizations rely on interview recordings. However, this takes a significant amount of resources. Therefore, some organizations are using paradata generated by the survey software, specifically timestamps, to try to detect when interviewers deviate from reading the question verbatim. However, there is no established method on how to use timestamps to detect question-reading deviations and little is known about the level of accuracy for the different methods currently used. This study evaluates the current methods used for detecting question-reading deviations using interview recordings and paradata from Wave 3 of the Understanding Society Innovation Panel. Using interview recordings allows a direct comparison of the different detection methods to how the interviewers actually administered the question and thus measure the accuracy of each detection method. Deviations are also coded for the extent (i.e., minor or major) and type of deviation. This analysis will give better insight on the scope and types of deviations interviewers are engaging in and practical guidance on how to best detect question-reading deviations.

Subjects

Computing and Survey Methodology

Links