Methods for Testing and Evaluating Survey Questionnaires
Contemporary Sociology: A Journal of Reviews, 2005
This collection of papers is a recent addition to the Wiley Series in Survey Methodology. This se... more This collection of papers is a recent addition to the Wiley Series in Survey Methodology. This series is considered by many, including myself, to be the definitive set of resources for practitioners of survey research. The current volume reviews practices in the pre-testing of survey questionnaires that have evolved in recent years. The book is organized by the editors into seven areas: Cognitive Interviewing; Supplements to Conventional Pretests (behavior coding, response latency analysis, respondent debriefing, and use of vignettes); Experiments (to evaluate alternative wording/ design); Statistical Modeling (applying statistical techniques to relate measurement error to faulty questions, discussion of item response theory, etc.); Mode of Administration (differences in pre-testing surveys administered by various modes): Special Populations (pre-testing surveys for establishments, children and adolescents, and multilingual populations); and Multi-Method Applications. The first section covers in considerable detail cognitive interviewing as a way to identify problem questions/sequences of questions. The selected articles provide a realistic assessment of the utility of cognitive interviewing in pre-testing survey questionnaires. The emphasis in this section, as well as throughout the volume, on the illustration of techniques and procedures creates a welcome “handbook” feel to the collection. Particularly useful are the various authors’ attempts to disentangle how cognitive interviewing practices, including probing techniques, affect the evaluation of question performance. Several of the authors raise and discuss issues concerning cognitive interviewing that are important to the survey practitioner but are not covered in-depth. Specifically, there is only some discussion of cost/benefits of cognitive interviewing as part of the pre-test process. Within the context of a large scale survey, the cost of cognitive interviewing is not prohibitive. However, with small scale surveys of local populations, extensive pre-testing, especially cognitive interviewing, is not cost effective. The fundamental question is whether or not the marginal gain in data quality merits the expenditure. Another issue touched on in the fifth article in this section but not explored is the roll of the survey client in the pre-test process. Perhaps the most important cognitive interview one might do is with the survey client. If the researcher fails to measure the client’s core concepts, pre-testing is not productive. What I concluded after reading the section is that cognitive interviewing can be a useful tool in many situations, but researchers have to be careful not to rely too heavily on a technique that uses information collected by a few interviewers from a few respondents to make decisions about questions to be used in a general survey. The second group of papers focuses on pre-test techniques that are sometimes used as supplements to conventional pretests. Behavior coding, response latency, respondent debriefing, and use of vignettes to evaluate the adequacy of questions are explored. Each paper in this section presents enough information for the practitioner to understand each of the techniques and then translate them to practice. Particularly useful are the papers on behavior coding and response latency. The comparison of the authors’ diagnostic approach to behavioral coding to other methods of assessing problematic questions clearly shows the advantages of their approach to traditional behavioral coding. The discussion of the costs/benefits of the diagnostic approach is appreciated. The third section reviews some examples of the use of experiments/field experiments in developing surveys. Pre-test experiments are not presented by the authors as alternatives to cognitive interviewing, behavior coding, and other methods of assessing questions. Each method/process produces different types of useful information. This section is a call to practitioners to design the pre-testing of questionnaires using the same rigor as any traditional study. They correctly observe that pre-tests are commonly done
The effect o f two important characteristics o f D illm an's Total Design M ethod f o r mail surv... more The effect o f two important characteristics o f D illm an's Total Design M ethod f o r mail surveys (personalized cover letters and last reminder by certified mail) were experimentally investigated. A sample from the general population o f the Netherlands was used. I f the Total Design M ethod is used the response in the Netherlands is comparable to the response in the United States (± 70%). Both personalization o f cover letter and the use o f certified mail is important; omission of either one results in a significantly lower response. Inleiding * De auteurs d a n k e n p r o f .d r. J. d e Jo n g-G ierv eld v o o r h a a r c o m m e n ta a r o p een ee rd e re versie v an d it m a nuscript. E en u itg e b re id o n d e rz o e k sv e rsla g is o p te v rag en b ij de ee rste a u te u r. ** De auteurs zijn v erb o n d e n a a n d e U n iv ersiteit v an A m ste rd a m .
Computer assisted data collection, data quality and costs; an annotated bibliography
Collecte de données par ordinateur, qualité des données et coûts - une taxonomie et bibliographie... more Collecte de données par ordinateur, qualité des données et coûts - une taxonomie et bibliographie annotée. Par des appels aux contributions publiés et des analyses de bases de données, l'auteur a repéré la littérature sur la collecte de données par ordinateur et la qualité des données pour construire une bibliographie annotée dans laquelle chaque entrée est codée par rapport à une taxonomie.
ABSTRACT Meta-analysis concerns the statistical integration of a larger number of results from em... more ABSTRACT Meta-analysis concerns the statistical integration of a larger number of results from empirical studies ...
The Design and Implementation of Mixed-mode Surveys
John Wiley & Sons, Inc. eBooks, Sep 28, 2018
In an optimal design for mixed‐mode surveys, be it within a country or across countries, there ar... more In an optimal design for mixed‐mode surveys, be it within a country or across countries, there are three phases that need attention. The first phase is the design phase: it is important to prevent mode measurement effects when designing the study. A richer source of additional data will help to achieve a better estimation of mode effects. After the data are collected, the next two phases are estimating the potential mode measurement effect due to different modes and adjusting for any such bias. Survey modes differ on several dimensions, for instance, interviewer‐administered versus self‐administered questionnaires and information transmission and communication. The use of well‐established and validated constructs and multiquestion scales across countries and modes helps achieving measurement equivalence countries and modes. Mixed‐mode designs and mode changes are often implemented for sound methodological reasons. Using a mixed‐mode design implies a careful trade‐off among coverage, nonresponse, measurement errors, and cost.
Journal of survey statistics and methodology, Jan 9, 2020
There is an ongoing debate in the survey research literature about whether and when probability a... more There is an ongoing debate in the survey research literature about whether and when probability and nonprobability sample surveys produce accurate estimates of a larger population. Statistical theory provides a justification for confidence in probability sampling as a function of the survey design, whereas inferences based on nonprobability sampling are entirely dependent on models for validity. This article reviews the
Computer assisted telephone interviewing, and to a lesser degree, computer assisted face-to-face ... more Computer assisted telephone interviewing, and to a lesser degree, computer assisted face-to-face interviewing, are by now widely used in survey research. Recently, self-administered forms of computer-assisted data collection, such as web surveys, have become extremely popular. Advocates of computer assisted interviewing (CAI) claim that its main advantages are improved data quality and lower costs. This paper summarizes what is currently known about computer assisted data collection methods. The emphasis is on data quality and the influence of technology on the respondent.
Bulletin de méthodologie sociologique, Apr 1, 2012
We still cannot look inside people’s heads. To measure emotions, opinions, evaluations, associati... more We still cannot look inside people’s heads. To measure emotions, opinions, evaluations, associations, and intentions, we have to ask people about these phenomena. Our data collection has changed over time, and many different methods are available. At present, Internet surveys are widely used in market research and are becoming an important data collection tool in universities. The main advantages of Internet surveys are the speed and cost reduction in comparison to more traditional data collection methods, the positive effect of self-administration, such as more privacy, use of more complex questionnaire routing, and in general a greater interactivity. Disadvantages are problems of Internet coverage of the general population, especially under-coverage of certain subgroups such as the elderly and the lower educated, and non-response. This contribution provides a review of existing studies comparing online surveys with more traditional data collection methods.
This study investigates the impact of different modes and tailor-made response enhancing measures... more This study investigates the impact of different modes and tailor-made response enhancing measures (TMREM)-such as bilingual interviewers with a shared ethnic background and translated questionnaires-on the measurement of substantive variables in surveys among minority ethnic groups in the Netherlands. It also provides insight into the ability to detect mode measurement effects of a recently developed method for disentangling mode measurement and mode selection effects, as well as into the tenability of the assumptions underlying this method. The data used in this study come from a large-scale survey design experiment among the four largest non-Western minority ethnic groups in the Netherlands comparing single-mode computer-assisted personal interviewing (CAPI) and sequential computerassisted web interviewing, computer-assisted telephone interviewing, and CAPI-MM. The number and intensity of the TMREM varied among the four
Thirty Years of Survey Methodology / Thirty Years of BMS
Bulletin de méthodologie sociologique, Oct 1, 2013
This article presents an overview of the history of survey methodology and survey statistics base... more This article presents an overview of the history of survey methodology and survey statistics based on an analysis of BMS issues since the start in October 1983. The author, member of the BMS Scientific Committee, discusses the changes in modes of data collection, the changing emphasis on different error sources in surveys, and trends in analysis and analysis tools. She reflects on recent developments in mixed-mode and Internet surveys, and ends with some thoughts on the future.
In den letzten Jahren hat die Schweiz eine starke Zunahme der Migrationsstrome erlebt, die mit ei... more In den letzten Jahren hat die Schweiz eine starke Zunahme der Migrationsstrome erlebt, die mit einer Veranderung des Bildungsniveaus der Einwandernden einherging. In der Vergangenheit waren diese neuen Realitaten aufgrund fehlender Daten schwer bezifferbar. Vorliegender Artikel beschreibt auf der Basis neuer Daten die Entwicklung der Zusammensetzung der Migrationsstrome in die Schweiz unter dem Aspekt des Bildungsniveaus und untersucht die Rolle des Arbeitsmarktes im Kontext dieser Migration. So hat sich die Zuwanderung hochqualifizierter Arbeitskrafte zwischen 1991 und 2014 mehr als verdoppelt. Insbesondere Migrantinnen und Migranten aus dem Vereinigten Konigreich, Nordamerika und Indien verfugen im Gegensatz zu Eingewanderten aus Portugal, Westafrika oder Sudamerika mehrheitlich uber eine Tertiarbildung. Und schliesslich erklart sich diese Zuwanderung hauptsachlich durch die Nachfrage auf dem Arbeitsmarkt, die mit einheimischen Arbeitskraften nicht befriedigt werden konnte.
Trends in household survey nonresponse: A longitudinal and international comparison
We first present an overview of international figures on nonresponse and their trend over time, d... more We first present an overview of international figures on nonresponse and their trend over time, distinguishing between noncontacts and refusals. In the second part we model differences between countries, using background variables on survey design and ...
Self-administered questionnaires have many advantages, especially when sensitive questions are as... more Self-administered questionnaires have many advantages, especially when sensitive questions are asked. However, paper self-administered questionnaires have a serious drawback: only relatively simple questionnaires can be used. Computer Assisted Self-Interviewing (CASI) can overcome these problems, and make it possible to use very complex self-administered questionnaires. CASI can take several forms, for instance, it can be a part of a personal (CAPI) interview where the interviewer hands over the computer to the respondent for specific questions. Another form is a computerized version of the mail survey: Disk-by-Mail. We have used both forms in an application for very special populations (primary school children, visually impaired young adults, and parents and children from multi problem families, in which professional guidance for the family was sought). This paper provides an introduction to computer assisted self-interviewing (CASI) and reviews the advantages and disadvantages of CASI with special attention to data quality. We discus the advantages of CASI when investigating special groups and topics and provide general advice on how to tailor standard CASI procedures for surveys of special groups.
Uploads
Papers by Edith de Leeuw