Qualitätssicherung bei der Datener hebung von international vergleichenden Umfragen am Beispiel des ESS - PDF

Description
Methoden Daten Analysen 2009, Jg. 3, Heft 2, S Data Collection Quality Assurance in Cross-National Surveys: The Example of the ESS Qualitätssicherung bei der Datener hebung von international

Please download to get full document.

View again

of 20
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Information
Category:

Letters

Publish on:

Views: 15 | Pages: 20

Extension: PDF | Download: 0

Share
Transcript
Methoden Daten Analysen 2009, Jg. 3, Heft 2, S Data Collection Quality Assurance in Cross-National Surveys: The Example of the ESS Qualitätssicherung bei der Datener hebung von international vergleichenden Umfragen am Beispiel des ESS Achim Koch, Annelies G. Blom, Ineke Stoop and Joost Kappelhof Abstract The significance of cross-national surveys for the social sciences has increased over the past decades and with it the number of cross-national datasets that researchers have access to. Cross-national surveys are typically large enterprises that demand dedicated efforts to coordinate the process of data collection in the participating countries. While cross-national surveys have addressed many important methodological problems, such as translation and the cultural applicability of concepts, the management of the data collection process has yet had little place in cross-national survey methodology. This paper describes the quality standards for data collection and their monitoring in the European Social Survey (ESS). In the ESS data are collected via faceto-face interviewing. In each country a different survey organisation carries out the data collection. Assuring the quality across the large number of survey organisations is a complex but indispensable task to achieve valid and comparable data. Zusammenfassung International vergleichende Umfragen haben in den vergangenen Jahrzehnten zunehmende Bedeutung in den Sozialwissenschaften erlangt. Diese Umfragen sind für gewöhnlich große Unterfangen, die gezielte Anstrengungen zur Koordinierung der Datenerhebung in den teilnehmenden Ländern erfordern. Probleme des Managements der Datenerhebung bei international vergleichenden Umfragen haben bislang jedoch nur wenig Aufmerksamkeit gefunden, im Unterschied etwa zu anderen methodischen Herausforderungen wie Fragen der Übersetzung oder der interkulturellen Übertragbarkeit von theoretischen Konzepten. Der vorliegende Beitrag beschreibt die Qualitätsstandards für die Datenerhebung und deren Überwachung im European Social Survey (ESS). Im ESS werden Daten in persönlich-mündlichen Interviews erhoben; in jedem Teilnehmerland ist ein anderes Umfrageinstitut mit der Feldarbeit betraut. Um valide und vergleichbare Daten zu erzielen, sind Maßnahmen zur Sicherung der Qualität der Datenerhebung über die große Zahl von Umfrageinstituten hinweg unverzichtbar. 220 Methoden Daten Analysen 2009, Jg. 3, Heft 2, S Introduction With growing globalisation the importance of cross-national data has increased and with it also the number of cross-national surveys (Kish 1994; Jowell 1998; Heath et al. 2005; Lynn et al. 2006). Cross-national surveys are large enterprises that demand considerable financial, human and infrastructural resources, at the country-level and the cross-national level. To assure reliable and valid measurement within each country as well as cross-country data comparability survey standards are specified and their implementation is monitored centrally. The participating countries are to adhere to the survey standards and provide proof of correct implementation. Cross-national surveys differ in the level of standardisation that they pursue. Whereas some surveys only specify a very limited set of survey standards (such as question wording and a minimum sample size), other surveys cover all aspects of the survey life-cycle. The present paper describes the quality standards for data collection and their monitoring in the European Social Survey (ESS). It is uncontested that social measurements like quantitative surveys need some kind of standardisation of methods and processes to provide reliable and valid data (Jowell 1998). This holds both for national and for cross-national surveys except for one important difference. National surveys usually have a single design (Lynn et al. 2006). This means that there is one sample design and one questionnaire is administered in a standard way by interviewers who have received the same training and instructions. In cross-national surveys, designs differ across countries due to differences in financial resources, legislation regarding the survey business, available sampling frames, the geographical dispersion of the population, languages, the experience and capability of survey organisations and survey practices (like the typical methods and content of interviewer training or the prevailing mode of interviewing) (Park/Jowell 1997; Smith 2007). Consequently, even in highly standardised cross-national surveys some aspects of the survey design will be implemented differently across countries. When differences in methods affect survey outcomes, comparisons across countries can be jeopardised, because observed cross-country differences may be mere methodological artefacts. If standardisation in methods leads to equivalent outcomes, cross-national surveys should therefore strive for perfect standardisation. However, for reasons mentioned above perfect standardisation is impossible. Moreover, occasionally the effect of methods can differ across countries. Skjåk and Harkness (2003) for instance argue that optimal modes of interview administration (face-to-face, telephone, self-completion, etc.) in one country may be quite Koch/Blom/Stoop/Kappelhof: Data Collection Quality Assurance problematic in others. Therefore, sometimes comparability of results may best be achieved by a deliberate variation in design. Such considerations seem quite plausible also with regard to response and nonresponse. For instance, in order to achieve similar response rates between countries it can be prudent to allow for the use of different types of respondent incentives across countries. Or, given differences between countries in at-home-patterns of their population, it may be advisable to accept different call schedules to achieve similarly low noncontact rates in all countries. '[T]he challenge is to identify which aspects of design need to be identical, which should be allowed (encouraged) to vary and within what parameters and which may be less important, in the sense that relevant characteristics of the survey data may be insensitive to variations in design.' (Lynn et al. 2006: 14f.) With regards to equivalence in probability sampling Lynn et al. (2006) argue that different sampling strategies may be the best way to achieve equivalent samples (see also Kish 1994; Häder/Lynn 2007); while equivalence of measurement may be best achieved by standardising question wording and mode of interview. For other aspects of survey design, such as data collection practices, the authors note that little is known yet about the effects of different design options. We look at quality assurance for data collection in cross-national surveys using the example of the ESS. The ESS is a biennial cross-national survey of social and political attitudes in Europe. Data are collected via face-to-face interviewing. 1 In the ESS standards for data collection are set by a Central Coordinating Team (CCT), which also produces guidelines, assists countries in preparing fieldwork, monitors the progress of fieldwork in all countries and evaluates the implementation processes. In each country a different survey organisation carries out the data collection. 2 Assuring the quality across such a large number of survey organisations is a complex but indispensable task to achieve valid and comparable data. We describe how the CCT of the ESS coordinates data collection in the more than 30 participating countries and how it tries to find a viable balance between standardisation and national adaptation. 1 Cross-national surveys often rely on face-to-face interviewing. Apart from the ESS, for instance also the Adult Literacy and Life Skills Survey, the Eurobarometer, the European and World Values Surveys, the Survey on Health, Ageing and Retirement in Europe and the World Mental Health Survey are conducted face-to-face. 2 A few ESS countries appoint local branches of globally acting groups like Ipsos, TNS or Gallup to carry out the ESS. In such a case, the national branches of global survey organisations usually act quite independently from each other. Some cross-national surveys (e. g. the Eurobarometer) subcontract the entire cross-national data collection to one global survey organisation. Here the cross-national coordination of the data collection is the task of the central office of the global survey organisation. 222 Methoden Daten Analysen 2009, Jg. 3, Heft 2, S We first provide basic background information on the ESS and describe how standards for data collection are set and monitored. Subsequently, outcomes of this approach for key data collection features in the first three rounds of the ESS are presented. We describe to what extent countries adhered to data collection standards and discuss reasons for deviations from these standards. The conclusion provides some final considerations. 2 The ESS: Basic Features, Aims and Organisation The ESS is an academically-driven social survey designed to chart and explain the attitudes, beliefs and behaviour patterns of its diverse populations (Jowell et al. 2007). In addition to monitoring and interpreting social change, the ESS also seeks to consolidate and improve cross-national quantitative measurements within Europe and beyond (O Shea et al. 2003). Since 2002 the survey has been fielded every two years and now, in its fourth round, it covers more than 30 countries. Each of the participating countries conducts approximately 2000 face-to-face interviews in each round, either as paper-and-pencil interviews (PAPI) or as computer assisted personal interviews (CAPI). The ESS questionnaire includes two main sections: a core module which remains relatively constant from round to round plus two or more rotating modules repeated at intervals. The core module monitors change and continuity in a wide range of social variables. The rotating modules focus on particular academic or policy concerns, like immigration or family, work and wellbeing. The average interview length is about 70 minutes. The ESS project is directed by the CCT, led by the Centre for Comparative Social Surveys at City University London (see Figure 1). The CCT is responsible for the design and coordination of the project. Its work is primarily funded by the European Commission. A multi-disciplinary team of researchers from seven European research institutes cooperates in the CCT. 3 Each partner institute has pre-specified and self-contained responsibilities, some of which continue throughout the project s life, others for shorter periods. The work comprises more than ten workpackages, including the general coordination and implementation of the project, sampling design, translation, fieldwork commissioning, piloting, archiving 3 Apart from City University, these institutes include NSD in Norway, GESIS in Germany, SCP in the Netherlands, the Universitat Pompeu Fabra in Barcelona, Spain, the University of Leuven in Belgium, and the University of Ljubljana in Slovenia. Koch/Blom/Stoop/Kappelhof: Data Collection Quality Assurance and dissemination. The seven institutes are also jointly responsible for overall quality control and quality assessment. Data collection and other national costs in each country are borne by national funding bodies. In each participating country the national funding agency appoints a National Coordinator (NC) and a survey organisation to implement the survey according to common ESS specifications. The NC and the survey organisation are responsible for the national implementation, including the sampling, translation, data collection, data editing and survey documentation. Figure 1 Organisational Structure of the ESS Specialist Advisory Groups Question Design Teams Scientific Advisory Board Funders Forum Methods Group Sampling Panel Translation Taskforce Central Coordinating Team National Coordinators and Survey Institutes Country Country Country The two key actors the CCT and NCs are supplemented by a network of overseeing and supporting groups: the Scientific Advisory Board, the Question Design Teams, the Sampling Panel, the Translation Taskforce, a methodological advisory board (the Methods Group) and a group representing the national funding bodies and the European Commission (the Funders Forum). 224 Methoden Daten Analysen 2009, Jg. 3, Heft 2, S Survey Standards in the ESS 3.1 Specification of Standards for Data Collection in the ESS Cross-national surveys vary in the balance of responsibilities at the cross-national and national levels. Given the large number of participating countries and aspired methodological rigour, the ESS needs a strong cross-national organ (the CCT) that stipulates the survey design and monitors quality. A standard specification designed by the CCT establishes the methods and procedures to be followed in all participating countries. Regarding the data collection process these Specifications for Participating Countries (European Social Survey 2001; 2003; 2005) cover three core areas: (1) the selection of a survey organisation, (2) data collection outcomes and (3) data collection procedures. In the following we summarise the ESS standards and describe the rationale behind them. Tables A1 and A2 in the appendix list the ESS data collection specifications distinguishing between required and recommended procedures. The tables also indicate the leeway for national adaptation for both groups of procedures. Selection of a Survey Organisation The ESS urges participating countries to contract the best European fieldwork organisations to ensure that its regular rounds of data collection are carried out to the same exacting standards (O Shea et al. 2003). Survey organisations to be appointed for the ESS must be capable of conducting national probability-based face-to-face surveys to the most rigorous standards. Furthermore, the specifications stipulate that, if necessary, the survey organisations should be willing to change their routine procedures and methods to ensure cross-national comparability. Accordingly, the ESS requires some flexibility on the part of survey organisations intending to field the ESS. The advantage of the country-wise selection of survey organisations is that NCs are best aware of the quality that organisations in their country can produce. However, it can be a challenge to get survey organisations to replace their traditional approach with ESS standards. This is an important task for the NCs. Section 3.3 demonstrates how this aim was pursued. Adherence to the ESS specifications is a prerequisite for each participating country and for each survey organisation selected to field the ESS. At times this may require a higher budget than is necessary for fielding a survey according to the usual standards in a country. Examples of fundamental changes to typical fieldwork practices come from France and Switzerland. In France the major challenge was to replace the traditional quota sampling by probability sampling, and in Switzerland the prevailing telephone mode had to be substituted by face-to-face interviewing. Koch/Blom/Stoop/Kappelhof: Data Collection Quality Assurance Data Collection Outcomes The ESS standards for data collection outcomes concern the sample size and the response and noncontact rate. The ESS specifications require a minimum effective sample size of 1500 interviews for each participating country based on a probability sample (Häder/Lynn 2007). Countries may use different sampling designs which may have a different effect on standard errors (independent from the size of the sample). To standardise the level of precision of results across countries the ESS prescribes an effective sample size, which takes account of the design effects associated with a country s sample design. The concept of an effective sample size operated in the ESS requires countries with geographically clustered samples to provide a higher number of completed interviews than countries using a simple random sample. Nonresponse is a major threat to sample surveys, since it decreases the net sample size and can lead to biased survey results (Groves/Couper 1998; Groves et al. 2002). In most Western countries response rates have been declining during the past decades (de Leeuw/de Heer 2002). The ESS specifies a minimum target response rate of 70 percent. When setting this target the CCT was aware that some countries would reach the target, while others would struggle. The CCT felt that specifying a target outcome rate to competing survey organisations would make the target a contractual obligation that the selected survey organisation must strive and budget for (Jowell et al. 2007). 4 The rationale was to maximise response rates in each country and to reduce variation in response rates across countries in order to optimise comparability. In addition to setting a target response rate the ESS limits the noncontact rate to three percent of the eligible sample. The reason for specifying a maximum noncontact rate was that this source of nonresponse can be easier controlled by insisting on certain design features (especially the number and timing of contact attempts) than the other major source of nonresponse, i. e. refusals (Groves/Couper 1998). Obviously, as regards nonresponse the ultimate goal should be to minimise nonresponse bias. However, minimising bias is even more difficult than enhancing 4 Of course, a certain response rate cannot be enforced. Individual target persons always have the right to refuse, may not be at home for a prolonged time or may not be able to participate in the survey because of illness, mental incapacities or language problems. If a survey organisation does not achieve the agreed upon rate, it has to be discussed whether additional fieldwork efforts and measures might be helpful when fielding the survey again in the future. Also a change of the survey agency might be considered. We should note that in the ESS only few countries included payment sanctions for not achieving the response rate target in their contract with the survey organisation. 226 Methoden Daten Analysen 2009, Jg. 3, Heft 2, S response rates. Nonresponse bias is estimate-specific and can vary substantially across variables within the same survey (Groves/Peytcheva 2008). Estimating nonresponse bias requires comparative auxiliary information for both respondents and non respondents, which cross-national surveys have trouble providing (Blom et al., forthcoming). Furthermore, a target nonresponse bias is extremely difficult to budget for in fieldwork reality. Nonresponse bias targets are demanding in national studies (for an interesting attempt see Schouten et al. 2009), and nearly impossible to use at least for the time being in cross-national multi-topic surveys like the ESS. Data Collection Procedures In order to achieve the specified data collection outcomes and to improve comparability across countries the ESS defines data collection procedures that each participating country needs to follow. These procedures include the mode of interview, maximum interviewer workloads and interviewer briefings, a set fieldwork period, interviewer calling schedules, the collection of contact data and quality control back-checks. Research has shown that the mode of data collection can affect survey results (Biemer/Lyberg 2003; Groves et al. 2004). Even within a country differential coverage, nonresponse and measurement errors across modes can cause mode effects; across countries the scope for differential errors are magnified. Consequently, the ESS collects its data in the same mode across all countries, namely by means of face-to-face interviews. For cross-national surveys face-to-face fieldwork offers several advantages over other modes including the best possible coverage of the target population and higher response rates (in most European countries). Furthermore, it is generally thought that the duration of a face-to-face int
Related Search
Similar documents
View more...
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks