An Embedded Experiment to Test Non-Response Follow-Up Strategies when using Electronic Questionnaires
|
|
- Vanessa Morris
- 5 years ago
- Views:
Transcription
1 WP. 27 ENGLISH ONLY UNITED NATIONS ECONOMIC COMMISSION FOR EUROPE CONFERENCE OF EUROPEAN STATISTICIANS Work Session on Statistical Data Editing (Oslo, Norway, September 12) Topic (iv): Using metadata and paradata to analyse the efficiency of editing processes An Embedded Experiment to Test Non-Response Follow-Up Strategies when using Electronic Questionnaires ABSTRACT I. Introduction Jeannine Claveau and Claude Turmelle, Statistics Canada 1. In, Statistics Canada launched a new project aiming at implementing Electronic Questionnaires (EQ) as the principal collection mode for business surveys. In preparation for the widespread adoption of EQ, an experimental design was conducted on seven current EQ surveys to evaluate different Non-Response Follow-Up (NRFU) strategies. The purpose of the study was to provide information to establish a standard collection follow-up strategy for EQ surveys. In essence, the study quantified the impact of using a telephone contact in comparison to using e- mail reminders for following-up with businesses that had not completed their survey. 2. This paper borrows heavily from Karaganis et al. (11) and Leung et al. (12). II. Overview of the UES Collection 3. The Unified Enterprise Surveys (UES) program consists of close to annual business surveys which are integrated in terms of content, collection and data processing. In 11, webbased EQ were developed and used to collect data for seven UES surveys, i.e. six surveys of the services industries and the head office survey. Before 11, only one of these surveys used EQ as a collection mode. 4. The UES collection process is performed in three steps. The first phase of the annual collection cycle for UES surveys consists of a telephone pre-contact conducted for new enterprises selected in the sample to confirm their contact information as well as the activity codes based on the North American Industry Classification System (NAICS). In 11, for the seven surveys with EQ, telephone pre-contact was conducted not only to confirm existing information, but also to obtain addresses of the businesses. Businesses were advised that collection would be done via electronic questionnaires and were asked to supply addresses. The only businesses assigned to paper questionnaires were those who adamantly refused EQ or 1
2 the ones who were not reached during pre-contact. For units who previously respond to EQ collection, a heads-up was sent instead to advise that their responses would once again be collected through EQ. 5. The second phase of the collection consisted of mailing out either paper questionnaires or invitations to the sampled units. The final phase of collection for the UES surveys consists of receiving completed questionnaires, imaging paper questionnaires, uploading both imaged and electronic questionnaires into the central collection system (Blaise) for edit verification and following-up with outstanding respondents (NRFU) and respondents whose questionnaire failed editing in Blaise (FEFU). UES collection is explained in more detail in Leung et al. (12). III. Methodology of the Experimental Design A. Embedded Experimental Design 6. An embedded balanced factorial design was used for this experiment. In the case of the UES it would not be feasible, due to the cost, to set up an experiment separate from the survey to study different non-response follow-up strategies. Conducting an experiment within a survey is normally cheaper than running a separate study, and with the recent economic climate this is an important consideration for us. 7. Recent research literature demonstrates that experiments embedded in ongoing sample surveys are particularly appropriate to investigate the effects of alternative survey methodologies on response behaviour (Van den Brakel and Renssen, 1998 and 5; Van den Brakel and Van Berkel, 2). Data collected using the current survey approach can be used to produce estimates and the complete set of data can be used for testing treatment differences. 8. An experiment within a survey can be thought of as a variation of a two-phase survey design, as illustrated in Figure 1. Since an embedded experimental design was used, testing if the treatments are significantly different requires the use of methods that take into account both the experimental phase and the sampling phase. The Wald test adjusted for the sample design as presented in Van den Brakel and Renssen (5) was used. Figure 1: Illustration of an experiment embedded in a sample survey. 2
3 9. In this paper, hypotheses are tested while taking into account the sample design, the experimental design, and the weighting procedure of the survey applied during estimation. Hypothesis testing can be done by using a simplified design-based Wald statistic by inserting design-based estimators for the subsample means and a covariance matrix of the contrasts between these subsample means. The design-based estimators for the subsample means assumes that each subsample can be considered as a two-phase sample, where the first order inclusion probabilities of the first phase sample are obtained from the sample design and the conditional first order inclusion probabilities of the second phase sample are obtained from the experimental design. B. Treatments. The set up of the experiment is illustrated in Figure 2. Note that the split between units receiving a paper questionnaire and an electronic questionnaire was not performed by randomization. Instead, it was determined by the respondents. Therefore, we cannot compare the results between paper and EQ collection. In fact, paper collection was composed of businesses that were not contacted during pre-contact and those who refused to use EQ. Thus, these businesses are more prone to be a non-response and are suspected to be qualitatively different from the units who went with the EQ collection mode. Figure 2: The set up of the experiment. Overall sample Telephone pre-contact or heads-up Paper questionnaire mail-out on Mar 18 EQ mail-out on Mar 28 NRFU by fax reminders and/or phone 1 then + Phone follow-up started + 3 s, 4 s, one every 2 weeks Phone follow-up started + 3 s, one per month 1 phone attempt then + 3 s, one every 2 weeks one per month 3
4 11. Businesses that participated in EQ collection were randomly assigned to one of four different follow-up treatments (combinations of telephone and reminders at different points throughout the collection period). The various treatments differ by the frequency of their follow-up attempts and whether or not respondents were called. Each treatment contained approximately 25% of the EQ sample. The randomization was done within each combination of survey, strata group and type of questionnaire. The three strata groups are: (1) small take-some stratum, (2) large take-some stratum and (3) take-all and must-take stratum. The type of questionnaire was identified by its length, i.e., long form or short form. It has to be noted that stratification was applied to six of the surveys while the Head Office survey was a census. As a result, we ended up with a randomized block design for service industries (surveys and strata used as blocks) and a completely randomized design for the Head Office survey. 12. On March 28, 11, each unit in Treatments 1 to 4 ( to ) received an invitation that contained a hyperlink and an access code for completing the survey online. Nonresponse follow-up started approximately one month into the collection, on April 26, 11. On that date, all outstanding units were sent to NRFU by a specified approach indicated for to. Each treatment had a different NRFU approach using a combination of reminders and phone attempts. NRFU stopped for a unit as soon as the questionnaire was returned to Statistics Canada. 13. The four treatments were designed as follows. Treatment 1 was designed to replicate the standard NRFU strategy. We sent a first reminder to the outstanding units then phone follow-up started. Three others reminders were also sent once per month. Treatment 2 was designed to test how far we could get with reminders only, i.e. what kind of response rates could we achieve without any phone follow-ups. We did not allow telephone follow-up, but if an enterprise called the help line at Statistics Canada, interviewers would call them back as needed. Because there was no phone follow-up at all, we decided to send reminders more often, once every two weeks. Treatment 3 was designed to measure the impact of doing the first followup attempt by phone rather than by . In this case, phone follow-up started before sending e- mail reminders, once per month. Treatment 4 was designed to combine both having the first follow-up attempt done over the phone and then switching to only reminders every two weeks. 14. The embedded experiment was to be conducted on live surveys, with the collected data to be used to produce usual estimates. This placed a significant constraint on the experiment. We had to ensure that the results at the end of collection would only be minimally affected. Therefore, it was decided to stop Treatment 2 and 4 on June and then to switch to a final blitz of phone follow-up on all outstanding units. This was decided because the potential risk to damage the final response rates was assumed higher for Treatment 2 and 4 since these 2 treatments did not benefit from continuous phone follow-up from the beginning. The experiment ended on August 1, 11 for Treatments and. Officially after this date, all non-response units were switched to follow-up by telephone, regardless of their treatment. In reality, telephone follow-up for Treatments 2 and 4 began on July 8. 4
5 IV. Results A. Descriptive Statistics 15. There were a total of 9,324 units sampled for these seven surveys. Among these units, 6,457 (approximately % of the overall sample) were assigned to EQ collection after precontact. These EQ units were then divided randomly into four treatments of approximately the same size. The four treatments had 1,615, 1,613, 1,615 and 1,614 units respectively. 16. Through follow-up of non-response, respondents could request a switch of a collection mode. Thus during collection, we observed 338 units switching from paper to EQ and 521 units switching from EQ to paper collection. Also, it was determined during collection that 1,98 units were out of scope. At the end of collection, the numbers of in scope units in the four treatments were 1,375, 1,353, 1,394 and 1,376, respectively. B. Return Rate 17. The progression over time of the number of questionnaires returned (cumulative unweighted return rate) for each of the four follow-up treatments is shown in Graph 1. This unweighted return rate based on the number of questionnaires returned is useful to assess the performance of a collection process, independently of the business size. The return rates were computed based on all in scope units at the end of collection. 18. All the events that happened on the key dates are indicated by the dotted lines labelled from E1 to E5. The first event, E1, was the beginning of the non-response follow-up. This first event happened at the end of April. Treatments 1 and 2 received an reminder and Treatments 3 and 4 received a phone attempt during that week. At the second event, E2 at the beginning of May, businesses subject to Treatments 2 and 4 received an reminder. At the third event, E3, at the end of May, all four treatments received an reminder. Event E4, at the beginning of June, was the last reminder for Treatment 2 and 4 before the end of experiment for these 2 treatments. Finally, event E5 represents the start of regular telephone NRFU for Treatment 2 and We can observe on the graph that, at the beginning of the non-response follow-up, the number of questionnaires returned for the four treatments were similar (between 14 to 16%). Two weeks later (on May 9, event E2 Graph 1), all the treatments had similar unweighted return rates (between 23 to 25%). But, after Treatments 2 and 4 received their second reminder, we observed a substantial increase in response. The lead in response was maintained until the end of May for Treatment 2 and until mid June for Treatment 4. It seems that having a reminder action every two weeks instead of every month is beneficial in obtaining responses. However, the reminder (event E4) on June 7 had little impact on response. The effectiveness of only sending e- mail reminders clearly diminishes over time. In fact, by June, after three months of collection, the businesses that received telephone follow-up showed much greater gains. But from another point of view, the reminders only treatment (Treatment 2) managed to obtain its first 45% of response simply by sending reminders every two weeks. By the end of collection, all test groups had similar return rates. The return rates for all four treatments were over % at the end of collection in mid-october. 5
6 4/11 5/9 6/6 7/4 8/1 8/29 9/26 Graph 1: Cumulative unweighted return rate observed on each day during collection for each treatment (March 28 to October 14, 11) E1 E2 E3 E4 E5 E1 -> April 26 /: /: phone attempt E2 -> May 9 /: E3 -> May 26 ///: E4 -> June 7 /: E5 -> July 8 /: NRFU started. When we looked at the seven surveys separately (Graphs 2 to 8), we observed more variability among the treatments for the unweighted return rates. For some surveys, the return rates for the treatments using continuous phone follow-up (Treatments 1 and 3) were slightly higher than those of the other two treatments (Graphs 2 and 3). But for other surveys, the treatments with no phone follow-up at the beginning (Treatments 2 and 4) seemed to give the highest rates (Graphs 4 and 7). Although some differences in the return rates were observed among the surveys, by the end of collection, the return rate for each treatment in each survey was always higher or equal to 74%. In some cases the percentage was over % and it sometimes even reached %. 21. We also looked at the graph of cumulative revenue weighted return rate. This revenue weighted return rate takes into account both the sample weight and the importance of the units in terms of business size. The revenue variable used to weight the return rate is the revenue variable available on the Business Register of Statistics Canada. This graph is similar to the unweighted rate in Graph 1. 6
7 Graph 2: Head Office Graph 3: Specialized Design Head Office Specialized Design Graph 4: Computer Services Graph 5: Architecture Computer Services Architecture 7
8 Graph 6: Engineering Graph 7: Accounting Engineering Accounting Graph 8: Consulting Consulting 8
9 C. NRFU Costs 22. The total cost of collection for UES surveys includes pre-contact, mail-out of questionnaires, NRFU, FEFU, capture and imaging, management and other tasks such as helpline, quality control monitoring, system testing, etc. In this paper, given the focus on NRFU strategies, we will concentrate on NRFU costs defined through the number of NRFU attempts and the length of such attempts among the four treatments. 23. A NRFU phone attempt is recorded in the Blaise System when an interviewer tries to contact the respondent at the time when a questionnaire has not yet been received by Statistics Canada. The start and end times of each attempt are recorded along with the outcome of the attempt. 24. The percentage of in-scope units having at least one NRFU call attempt by the end of collection (October 14) for each treatment is presented in Graph 9. As seen from Graph 1, approximately 15% of the units in each treatment returned their questionnaire by April 26. Only the remaining outstanding units were eligible for NRFU. 25. Treatment 2 had the smallest proportion of units getting the NRFU attempt, because outstanding units in Treatment 2 received only reminders in the first three months of the experiment. Telephone follow-up did not start until July 8 for this treatment. Note that some units in Treatment 2 received a call during the experiment because it was agreed at the beginning that if an enterprise called the help line at Statistics Canada, interviewers would call them back as needed. 26. Treatments 3 and 4 had the highest number of units with at least one phone attempt due to the set up of the experiment: outstanding units in these two treatments received a phone attempt during the week of April 26. For Treatment 1, the outstanding units did not receive any telephone follow-up until after receiving the first reminder. Thus, the percentage of Treatment 1 units with phone attempts was slightly lower than for Treatments with the initial follow-up attempt done over the phone. Graph 9: Percentage of in scope units that received at least one NRFU call attempt by the end of collection, October 14, The average number of call attempts for each in scope unit in the four treatments as of June and October 14 are presented in Graph. Recall that a maximum of five NRFU phone attempts was allowed before a unit was considered as a final non-response unit. Recall also that not all units were eligible for NRFU (as they might have submitted their questionnaire by the time NRFU started) and even if they were, not all outstanding units were contacted by phone due to set up of the experiment. As of June 9
10 Number of attempts, Treatment 2 was the only treatment that received reminders only. It is therefore expected that Treatment 2 would have a very low average number of attempts. The average started to pick up after July 8 since all outstanding units in the four treatments received telephone follow-up. By the end of collection, the average number of attempts for Treatment 2 was still the lowest, while for the other three treatments, the averages were similar. As expected, Treatment 3 had the highest average number of NRFU attempts as it included both phone follow-up throughout the experiment as well as the initial phone follow-up attempt on April 26. It also appeared that the effect of doing follow-ups and doing the first attempt over the phone are similar, in terms of average number of NRFU call attempts, as illustrated by Treatments 1 and 4. Graph : Average number of NRFU call attempts for each in scope unit in each treatment (on June and on October 14, 11) Average number of NRFU call attempts for each in scope unit June October 14.3 Treatment 1 Treatment 2 Treatment 3 Treatment As mentioned before, the start and end times for each attempt are recorded in the Blaise System. We are then able to compute the duration of the attempts, also known as Time Per Unit (TPU). One of the goals of this experiment was to find the most efficient non-response follow-up strategy, i.e., the one producing the best return rate at the lowest cost. Based on descriptive statistics, Treatments 1 and 3 were the treatments with the most intensive follow-up: they received reminders and telephone follow-up from the end of April. Consequently, they were also the most expensive. Treatment 4 was less expensive than Treatments 1 and 3 because there was no telephone follow-up until July 8, but there was one phone attempt at the beginning of NRFU. The least expensive strategy was Treatment 2, in which only reminders were used at the beginning and there were no telephone attempts or follow-up until July 8. The scatter plots of the unweighted return rate versus the TPU for non-response follow-up on June and October 14 are shown in Graph 11. Since it is more costly to perform telephone follow-up than to send e- mail reminders (which is an automated process), a treatment was considered more costly if the TPU on NRFU was higher. Therefore, from the scatter plots, we observed that Treatment 2 was able to give similar return rates with lower cost on phone follow-up (around 11 minutes by the end of collection), compared to (around 15 minutes) for the other three treatments. The difference represents approximately 25% cost reduction compared to phone-only NRFU (excluding overhead cost).
11 Graph 11: Scatter plots of unweighted return rate versus the average duration of NRFU attempts (TPU) (on June and on October 14, 11) D. Analysis of Variance 29. Analyses of variance on return and response rates, number of NRFU attempts and time spent per unit (TPU) on non-response follow-up and failed edit follow-up were performed to compare the different treatments. Recall that the Head Office Survey is a census where each unit has the same sampling weight, while for Services surveys a stratified sample is selected. Because of the difference in the sampling designs, the analysis of variance tests were run separately for Head Office and for Services surveys. For tests on Head Office units, the F-Test in the SAS procedure PROC GLM was used. For Services surveys, the Wald Test adjusted for design (Van den Brakel and Renssen, 5) was used.. In Table 1, the results of the analysis of variance for Services surveys show that the unweighted return rate was significantly different between the four treatments on June. Unweighted response and revenue weighted rates were not significantly different on June. At the end of collection, there were no significant differences between the four treatments in terms of return and response rates. For the Head Office survey, unweighted return and response rate were not significantly different between treatments on both June and October 14 dates. 31. On the other hand, the number of telephone follow-up attempts and the TPU were found to be significantly different. When we looked at the seven surveys separately, the number of telephone followup attempts and TPU were always lower for the reminders only treatment (Treatment 2). 32. Analyses of variance on the number of NRFU attempts and time spent per unit (TPU) on failed edit follow-up were performed to compare the different treatments. For all surveys, neither variable was significantly different between treatments. 11
12 Table 1: Testing treatment differences on June and on October 14 Main Effects: = = = Head Office Jun Oct 14 p-value p-value Services Jun Oct 14 p-value p-value Revenue weighted return rate Revenue weighted response rate Unweighted return rate Unweighted response rate NRFU attempts < <.1 <.1 FEFU attempts NRFU TPU < <.1 <.1 FEFU TPU V. Conclusions 33. From a collection perspective, the design of experiment showed that significant response can be obtained without making phone calls for the first two or three months of collection for annual surveys. In addition, more frequent reminders are beneficial for obtaining responses. The strategy consisting of reminders every two weeks at the beginning of collection succeeded in obtaining the first 45% of response at lower cost. On the other hand, there is a point in time when it becomes necessary to start telephone follow-up, but starting this process later does not appear to negatively impact the final survey response rates. References Karaganis, M., Fox, K., Claveau, J., Leung, J. and Lin, W. (11). Embedded Experiment for Nonresponse Follow-up Methods of design of Electronic Questionnaire Collection. Proceedings of Statistics Canada Symposium 11. Leung. J., Claveau, J., Turmelle, C., Fox, K., Lin, W. and Karaganis, M. (12). Experimental Design for Non-response Follow-up of an Electronic Questionnaire Survey. Proceedings of 12 Federal Committee on Statistical Methodology Research Conference. Van den Brakel, J.A. and Renssen, R.H. (1998). Design and Analysis of Experiments Embedded in Sample Surveys. Journal of Official Statistics, 14(3), Van den Brakel, J.A. and Renssen, R.H. (5). Analysis of experiments embedded in complex sampling designs. Survey Methodology, 31(1), 23-. Van den Brakel, J.A. and Van Berkel, C.A.M. (2). A design-based analysis procedure for twotreatment experiments embedded in sample surveys. An application in the Dutch labor force survey. Journal of Official Statistics, 18(2),
Using Paradata to Understand Business Survey Reporting Patterns
Using Paradata to Understand Business Survey Reporting Patterns Eric B. Fink Joanna Fane Lineback November 5, 2013 2013 Federal Committee on Statistical Methodology Research Conference Any views expressed
More informationResponsiveness and Representativeness in an Establishment Survey of Manufacturers
Responsiveness and Representativeness in an Establishment Survey of Manufacturers Eric B. Fink Joanna Fane Lineback October 2, 2014 2014 International Total Survey Error Workshop Any views expressed are
More informationMIXED MODE IN THE DATA COLLECTION OF SBS STATISTICS WITHIN STATISTICS SWEDEN
Distr. GENERAL 03 September 2013 WP 24 ENGLISH ONLY UNITED NATIONS ECONOMIC COMMISSION FOR EUROPE CONFERENCE OF EUROPEAN STATISTICIANS Seminar on Statistical Data Collection (Geneva, Switzerland, 25-27
More informationStatistical Design and Estimation taking account of Quality
Statistical Design and Estimation taking account of Quality Chris Skinner NTTS, 23 February 2011 Quality Quality as accuracy Aim: quality improvement 2 Quality Context despite increasing threats to quality
More informationBelize 2010 Enterprise Surveys Data Set
I. Introduction Belize 2010 Enterprise Surveys Data Set 1. This document provides additional information on the data collected in Belize between August and October 2011 as part of the Latin America and
More informationThe Solomon Islands 2015 Enterprise Surveys Data Set
The Solomon Islands 2015 Enterprise Surveys Data Set I. Introduction This document provides additional information on the data collected in Solomon Islands between September 2015 and May 2016. The objective
More informationThis lecture will cover. Quantitative Research When to use Quantitative Research Quantitative Research Methods
Market Research Lecture This lecture will cover Quantitative Research When to use Quantitative Research Quantitative Research Methods Survey Methods Observational Research Quantitative research About measurement
More informationGlossary of Research Terms
Glossary of Research Terms January 2001 fact sheet I 2/11 Ad hoc Single surveys designed for a specific research purpose, as opposed to continuous, regularly repeated, or syndicated surveys. Advertising
More informationField Reengineering of the Canadian Census
Field Reengineering of the Canadian Census International Conference on Census Methods July 31 st, 2014 Outline Recent History of the Canadian Census Canadian Context Census Field Infrastructure Staffing
More informationEffects of Incentive Amount and Type on Web Survey Response Rates
Effects of Incentive Amount and Type on Web Survey Response Rates Jared Coopersmith 1, Lisa Klein Vogel 1, Timothy Bruursema 1, Kathleen Feeney 1 1 Mathematica Policy Research, 1100 1 st Street, NE, Washington,
More informationThe Bahamas 2010 Enterprise Surveys Data Set
I. Introduction The Bahamas 2010 Enterprise Surveys Data Set 1. This document provides additional information on the data collected in the Bahamas between April 2011 and August 2011 as part of the Latin
More informationEthiopia - Socioeconomic Survey
Microdata Library Ethiopia - Socioeconomic Survey 2013-2014 Central Statistics Agency of Ethiopia (CSA) - Ministry of Finance and Economic Development, Living Standards Measurement Study Integrated Surveys
More informationSurvey Statistician to provide assistance for the Randomized rural household survey Scope of Work (SOW)
AgResults Kenya On-Farm Storage Pilot Survey Statistician to provide assistance for the Randomized rural household survey Scope of Work (SOW) 1. Consultant Name: TBD 2. Period of Performance: TBD 3. Level
More information1 PEW RESEARCH CENTER
1 Methodology This report is drawn from a survey conducted as part of the American Trends Panel (ATP), a nationally representative panel of randomly selected U.S. adults living in households recruited
More informationRecent Developments in Assessing and Mitigating Nonresponse Bias
Recent Developments in Assessing and Mitigating Nonresponse Bias Joanna Fane Lineback and Eric B. Fink 1 U.S. Census Bureau, Washington, D.C. 20233 Abstract In this paper, we address recent developments
More informationElectronic Data Collection in Accommodation Statistics
Electronic Data Collection in Accommodation Statistics Juha-Pekka Konttinen 1, Maria Velasco Gimeno 2 1 Statistics Finland, e-mail: juha-pekka.konttinen@stat.fi 2 INE Spain, e-mail: mrvelasco@ine.es Abstract
More informationUse of Paradata to Improve Survey Operations for Household and Business Surveys
Use of Paradata to Improve Survey Operations for Household and Business Surveys Operations Research and Process Improvement Australian Bureau of Statistics Introduction Australian Statistician MDMD MIG
More informationI. Survey Methodology
I. Survey Methodology The Elon University Poll is conducted using a stratified random sample of households with telephones and wireless telephone numbers in the population of interest in this case, citizens
More informationThe Common Editing Strategy and the Data Processing of Business Statistics Surveys
1 WP. 16 ENGLISH ONLY UNITED NATIONS STATISTICAL COMMISSION and ECONOMIC COMMISSION FOR EUROPE CONFERENCE OF EUROPEAN STATISTICIANS Work Session on Statistical Data Editing (Ljubljana, Slovenia, 9-11 May
More informationThe implementation of tools to support the data quality of the Business Register at Statistics Canada
The implementation of tools to support the data quality of the Business Register at Statistics Canada Mario Ménard, Canada Abstract Business Registers contains a vast array of information concerning enterprises.
More informationChapter3. Methodology of Study. Conceptual Framework. The conceptual framework for my research is based on the extensive interviews done.
Chapter3 Methodology of Study Conceptual Framework The conceptual framework for my research is based on the extensive interviews done. Influencing variables: Information Security Experience Knowledge on
More informationThe Trinidad and Tobago 2010 Enterprise Surveys Data Set
I. Introduction The Trinidad and Tobago 2010 Enterprise Surveys Data Set 1. This document provides additional information on the data collected in Trinidad and Tobago between March 2011 and September 2011as
More informationSample: n=2,252 people age 16 or older nationwide, including 1,125 cell phone interviews Interviewing dates:
Survey questions Library Services Survey Final Topline 11/14/2012 Data for October 15 November 10, 2012 Princeton Survey Research Associates International for the Pew Research Center s Internet & American
More informationImproving the Labor Force Questions in the American Community Survey: The Results of the 2006 ACS Content Test 1
Improving the Labor Force Questions in the American Community Survey: The Results of the 2006 ACS Content Test 1 David A. Raglin and Kelly A. Holder David Raglin, Decennial Statistical Studies Division,
More informationExperiences in Implementing a Responsive Collection Design (RCD) for Blaise CATI Social Surveys
Experiences in Implementing a Responsive Collection Design (RCD) for Blaise CATI Social Surveys Éric Joyal, François Laflamme, Statistics Canada Over the past few years, paradata research has focused on
More informationUsing Paradata to Actively Manage Data Collection Survey Process
Using Paradata to Actively Manage Data Collection Survey Process François Laflamme 1, Mike Maydan 2, Andrew Miller 3, 1 Statistics Canada, Business Survey Methods Division, francois.laflamme@statcan.gc.ca
More informationMixed mode and web data collection
Mixed mode and web data collection Experiences in the ESS Annemieke Luiten Contents ESSnet on web and mixed mode data collection CBS and other experiences with web and mixed mode data collection How to
More informationA SYSTEMATIC AND STRATEGIC SYSTEM FOR DATA COLLECTION MANAGEMENT AT STATISTICS NORWAY
Distr. GENERAL 8 October 2012 WP. 19 ENGLISH ONLY UNITED NATIONS ECONOMIC COMMISSION FOR EUROPE CONFERENCE OF EUROPEAN STATISTICIANS Seminar on New Frontiers for Statistical Data Collection (Geneva, Switzerland,
More informationBangladesh - CGAP Smallholder Household Survey 2016, Building the Evidence Base on The Agricultural and Financial Lives of Smallholder Households
Microdata Library Bangladesh - CGAP Smallholder Household Survey 2016, Building the Evidence Base on The Agricultural and Financial Lives of Smallholder Households Jamie Anderson - Consultative Group to
More informationDesign, Implementation & Analysis of Innovation Surveys
Design, Implementation & Analysis of Innovation Surveys With a note on presenting results Anthony Arundel UNU-MERIT, The Netherlands Survey design and implementation 1 Tampere 200 Outline Part 1 How to
More informationBenin Indicator Survey Data Set
Benin Indicator Survey Data Set 1. Introduction 1.1. This document provides additional information for the Indicator Survey data collected in Benin from 18 th May and 30 th September 2009 as part of the
More informationAsia and Pacific Commission on Agricultural Statistics
October 2012 Asia and Pacific Commission on Agricultural Statistics Twenty-fourth Session Da Lat, Viet Nam, 8-12 October 2012 Agenda Item 7 United States- Agriculture Census and Surveys 2012 United States
More informationLecture 10. Outline. 1-1 Introduction. 1-1 Introduction. 1-1 Introduction. Introduction to Statistics
Outline Lecture 10 Introduction to 1-1 Introduction 1-2 Descriptive and Inferential 1-3 Variables and Types of Data 1-4 Sampling Techniques 1- Observational and Experimental Studies 1-6 Computers and Calculators
More informationIncreasing your profitability with BitTitan migration solutions
Increasing your profitability with BitTitan migration solutions Table of Contents Which BitTitan tool is right for your project?... 3 UserActivation... 3 MigrationWiz with DeploymentPro... 3 UserActivation:
More informationAnnamaria Bianchi 1 & Barry Schouten 2
Annamaria Bianchi 1 & Barry Schouten 2 1 DSAEMQ - University of Bergamo, Italy 2 Statistics Netherlands & Utrecht University 15 June 2017 What are Adaptive Survey Designs? Ingredients Web mode as a possible
More informationPersonal interview questionnaire suitable when open ended questions are asked
Chapter 4 Primary Data Collection Data Collection Methods for Quantitative Research Surveys The biggest advantage of the survey method is that at a time a lot of information(or data) about a single respondent
More informationCOMMUNITY INNOVATION SURVEY (CIS)
Standard Report on Methods and Quality for COMMUNITY INNOVATION SURVEY (CIS) Published as: Innovation in Irish Enterprises This documentation applies to the reporting period: 2012 2014 inclusive Last edited:
More informationPRACTICE QUESTIONS FOR CH 6 INTRODUCTION ANDD COLLECTION OF NUMERICAL DATA FOR PIPFA Exercise 1(p 171): - Look through the following list of surveys
PRACTICE QUESTIONS FOR CH 6 INTRODUCTION ANDD COLLECTION OF NUMERICAL DATA FOR PIPFA Exercise 1(p 171): - Look through the following list of surveys and decide whether each is collecting data on attributes,
More information1 PEW RESEARCH CENTER
1 Methodology The American Trends Panel (ATP), created by Pew Research Center, is a nationally representative panel of randomly selected U.S. adults recruited from landline and cellphone random-digit-dial
More informationDeveloping Response Metrics for the Economic Census 1
Developing Response Metrics for the Economic Census 1 Joanna Fane Lineback, Broderick Oliver, and Diane K. Willimack U.S. Census Bureau U.S. Department of Commerce, Washington, D.C. 20233 1. Introduction
More informationMAXIMISING THE OUTCOME OF SURVEYS. Natalija Bryžachina, Principal Auditor National Audit Office of Lithuania 8 October 2014
MAXIMISING THE OUTCOME OF SURVEYS Natalija Bryžachina, Principal Auditor National Audit Office of Lithuania 8 October 2014 WHAT IS A SURVEY? Survey a method of collecting standardised information from
More informationStrategic CSAT: Driving Strategic Outputs From Satisfaction Research
Report Summary Findings: Outsourcing customer satisfaction research is common but not universal: 54% of large scale enterprises fully outsource their CSAT programmes According to marketing decision makers,
More informationThe Central African Republic 2011 Enterprise Surveys Data Set
I. Introduction The Central African Republic 2011 Enterprise Surveys Data Set 1. This document provides additional information on the data collected in Central African Republic between June 2011 and July
More informationSurvey Expert to provide assistance for the Randomized rural household survey Scope of Work (SOW)
AgResults Kenya On-Farm Storage Pilot Survey Expert to provide assistance for the Randomized rural household survey Scope of Work (SOW) 1. Consultant Name TBD 2. Period of Performance TBD 3. Level of Effort
More informationResponsive design for household surveys: tools for actively controlling survey errors and costs
J. R. Statist. Soc. A (2006) 169, Part 3, pp. 439 457 Responsive design for household surveys: tools for actively controlling survey errors and costs Robert M. Groves and Steven G. Heeringa University
More informationThe Application of Blaise III to the Israel Labour Force Survey
The Application of Blaise III to the Israel Labour Force Survey Edith Noy and Gad Nathan, Central Bureau of Statistics, Jerusalem, Israel 1. Introduction The Central Bureau of Statistics has only recently
More informationBUSS1020 Quantitative Business Analysis
BUSS1020 Quantitative Business Analysis Week 1 - Introduction and Collecting Data Process of statistical analysis 1. Define the objective, and understand the data we need to collect. 2. Collect the required
More informationPlans for the 1991 Redesign of the Canadian Labour Force Survey
Plans for the 1991 Redesign of the Canadian Labour Force Survey D. Drew, J. Gambino, E. Akyeampong and B. Williams, Statistics Canada J. Gambino, Social Survey Methods Division, 16-A, R.H. Coats Building,
More informationThe Egypt 2016 Enterprise Surveys Data Set
The Egypt 2016 Enterprise Surveys Data Set I. Introduction This document provides additional information on the data collected in Egypt between October 2016 and April 2017. The objective of the Enterprise
More informationInternational Program for Development Evaluation Training (IPDET)
The World Bank Group Carleton University IOB/Ministry of Foreign Affairs, Netherlands International Program for Development Evaluation Training (IPDET) Building Skills to Evaluate Development Interventions
More informationPalestinian Central Bureau of Statistics Economic Statistics Directorate
Palestinian Central Bureau of Statistics Economic Statistics Directorate The Services Survey 2002 User s Guide Raw Data Table of Contents Concepts and Definition Survey Questionnaire Data Set linkage Target
More informationChapter 12. Sample Surveys. Copyright 2010 Pearson Education, Inc.
Chapter 12 Sample Surveys Copyright 2010 Pearson Education, Inc. Background We have learned ways to display, describe, and summarize data, but have been limited to examining the particular batch of data
More informationOff-Campus Work-Study Employer JobX Training Guide
Off-Campus Work-Study Employer JobX Training Guide Student Employment Services Training Topics: 1. Introduction 2. Login 3. Add a job 4. Edit a job 5. Review on-line applications 6. Send Hiring Correspondence
More informationNext Gen Web Solutions Student Employment. Employer Training Template
Next Gen Web Solutions Student Employment Employer Training Template Student Employment JobX + TimesheetX = Total Solution JobX helps schools automate the job posting, hiring, and reporting process for
More informationForecasting for Short-Lived Products
HP Strategic Planning and Modeling Group Forecasting for Short-Lived Products Jim Burruss Dorothea Kuettner Hewlett-Packard, Inc. July, 22 Revision 2 About the Authors Jim Burruss is a Process Technology
More informationMarketing Research to Support the Stage Gate New Product Development Process
Marketing Research to Support the Stage Gate New Product Development Process Agile. Iterative. Informed. Fast. These are hallmarks of today s effective new product development, with the goal of establishing
More information25 th Meeting of the Wiesbaden Group on Business Registers - International Roundtable on Business Survey Frames. Tokyo, 8 11 November 2016
25 th Meeting of the Wiesbaden Group on Business Registers - International Roundtable on Business Survey Frames Tokyo, 8 11 November 2016 Tammy Hoogsteen Statistics Canada Session No. 7 Globalization and
More informationMonitoring & Evaluation Guidelines
Identifying M&E Indicators Monitoring & Evaluation Guidelines United Nations World Food Programme Office of Evaluation What is an Indicator 4 How to check the Link between the Operation Design and the
More informationExamples of cost calculation for various mixed mode approaches: Cost drivers and their effectiveness
Examples of cost calculation for various mixed mode approaches: Cost drivers and their effectiveness Stefan Klug and Birgit Schrödle DemoSCOPE Research & Marketing AG, Switzerland 1 Mixed mode in (commercial)
More informationSession 4.2: Sampling Designs for Horticulture Surveys
Second Regional Training Course on Sampling Methods for Producing Core Data Items for Agricultural and Rural Statistics Module 4: Sampling Methods for Horticultural Surveys Session 4.2: Sampling Designs
More informationNational Quality Report SWEDEN 2005
National Quality Report SWEDEN 2005 The objective of this paper is to give an overall picture of the quality of the Swedish data for the Structural Business Statistics (SBS). The report consists of three
More informationSurvey frame, sample size and sample design. UN Handbook chapter 4 Sidney Vergouw
Survey frame, sample size and sample design UN Handbook chapter 4 Sidney Vergouw Outline of this presentation Some words on the handbook General remarks on precision, sample methods and sample size Consumer
More informationDisplaying Bivariate Numerical Data
Price ($ 000's) OPIM 303, Managerial Statistics H Guy Williams, 2006 Displaying Bivariate Numerical Data 250.000 Price / Square Footage 200.000 150.000 100.000 50.000 - - 500 1,000 1,500 2,000 2,500 3,000
More informationDistinguish between different types of numerical data and different data collection processes.
Level: Diploma in Business Learning Outcomes 1.1 1.3 Distinguish between different types of numerical data and different data collection processes. Introduce the course by defining statistics and explaining
More information2009 First Nations Client Survey
TOP-LINE REPORT 2009 First Nations Client Survey Prepared for: Prepared by: Malatest & Associates Ltd. CONTENTS SECTION 1: INTRODUCTION..... 1 1.1 Project Background... 1 1.2 Survey Objectives... 1 1.3
More informationDirect Monitoring and Measuring of Vehicle Utilization and Efficiency
Direct Monitoring and Measuring of Vehicle Utilization and Efficiency 2 nd International Transport Forum Statistical Meeting Presentation by Transport Canada April 1, 2015 Outline Objective and background
More informationPrepared by: Chintan Turakhia, Jonathan Best, and Jennifer Su 1 Braxton Way Suite 125 Glen Mills, PA 19342
Methods Report for Vanderbilt University s: Tennessee Poll Spring 2018 S u r v e y o f T N R e g i s t e r e d V o t e r s A g e 1 8 + M a y 1 0, 2 0 1 8 Prepared by: Chintan Turakhia, Jonathan Best, and
More informationIntroduction and Background
Introduction and Background In 1973 the National Center for Health Statistics expanded its National Nursing Home Survey (NNHS) to collect data on the cost to facilities of providing long -term health care.
More informationWelcome to NHTSA s public webinar of the redesign of its NASS. This webinar is to provide information and updates on the redesign of NHTSA s National
1 Welcome to NHTSA s public webinar of the redesign of its NASS. This webinar is to provide information and updates on the redesign of NHTSA s National Automotive Sampling System. We will mostly go over
More informationBEFORE THE PUBLIC SERVICE COMMISSION OF THE STATE OF UTAH ROCKY MOUNTAIN POWER. Exhibit Accompanying Direct Testimony of Robert M.
Exhibit RMP (RMM-2) BEFORE THE PUBLIC SERVICE COMMISSION OF THE STATE OF UTAH ROCKY MOUNTAIN POWER Exhibit Accompanying Direct Testimony of Robert M. Meredith Load Research Sampling Procedures January
More informationTender EACI/CIP/10/01 "European IPR Helpdesk"
Tender EACI/CIP/10/01 "European IPR Helpdesk" QUESTIONS SENT TO MAILBOX EACI-IPR-HELPDESK@EC.EUROPA.EU AND REPLIES Newly published questions and answers on 24 th t September: 16, 17,18,19,20 Question 1
More informationTopic 2 Market Research. Higher Business Management
Topic 2 Market Research Higher Business Management 1 Learning Intentions / Success Criteria Learning Intentions Market research Success Criteria By end of this topic you will be able to explain: the different
More informationFinScope Methodology
FinScope Methodology 1. FinScope Surveys The FinScope survey is a research tool developed by FinMark Trust. It is a nationally representative survey of how people source their income, and how they manage
More informationArchives of Scientific Psychology Reporting Questionnaire for Manuscripts Describing Primary Data Collections
(Based on APA Journal Article Reporting Standards JARS Questionnaire) 1 Archives of Scientific Psychology Reporting Questionnaire for Manuscripts Describing Primary Data Collections JARS: ALL: These questions
More informationUnified Communications Deployment Tool
Unified Communications Deployment Tool Shadan Saniepour Esfahani Cisco Systems Advanced Services Unified Communications Practice ssaniepo@cisco.com Talal Siddiqui Cisco Systems Advanced Services Unified
More informationRequirements Analysis and Design Definition. Chapter Study Group Learning Materials
Requirements Analysis and Design Definition Chapter Study Group Learning Materials 2015, International Institute of Business Analysis (IIBA ). Permission is granted to IIBA Chapters to use and modify this
More information9-3. Learning Objectives
9-1 Surveys Learning Objectives 9-3 Understand... The process for selecting the appropriate and optimal communication approach. Factors affect participation in communication studies. Sources of error in
More information9-3. Learning Objectives
9-1 Surveys Learning Objectives 9-3 Understand... The process for selecting the appropriate and optimal communication approach. Factors affect participation in communication studies. Sources of error in
More informationChapter 3 Sales forecasting
Chapter 3 Sales forecasting Nature and purpose of sales forecasting It would not be hard to be a successful business person if you had a crystal ball and could look into the future. If you knew which products
More informationMAS187/AEF258. University of Newcastle upon Tyne
MAS187/AEF258 University of Newcastle upon Tyne 2005-6 Contents 1 Collecting and Presenting Data 5 1.1 Introduction...................................... 5 1.1.1 Examples...................................
More informationMANAGEMENT ACCOUNTING 2 MODULE CODE: ACCT08004
School of Business & Enterprise Paisley & Hamilton Campus Session 014-015 Trimester 1 MANAGEMENT ACCOUNTING MODULE CODE: ACCT08004 Date: 16 th January 015 Time: 1000-100 Answer THREE questions Question
More informationENVIRONMENTAL FINANCE CENTER AT THE UNIVERSITY OF NORTH CAROLINA AT CHAPEL HILL SCHOOL OF GOVERNMENT REPORT 3
ENVIRONMENTAL FINANCE CENTER AT THE UNIVERSITY OF NORTH CAROLINA AT CHAPEL HILL SCHOOL OF GOVERNMENT REPORT 3 Using a Statistical Sampling Approach to Wastewater Needs Surveys March 2017 Report to the
More informationKey Words: survey costs; automation
AUTOMATED DATA COLLECTION STRATEGIES FOR THE COVERED EMPLOYMENT AND WAGES (CEW) PROGRAM Michael A Searson, U.S. Bureau of Labor Statistics Suite 4840, 2 Massachusetts Avenue, NE, Washington, DC 20212 Key
More informationA Canadian Commodity Flow Survey
A Canadian Commodity Flow Survey Statistics Canada and Transport Canada Transportation Research Board Commodity Flow Survey Workshop The Keck Center of the National Academies Washington, D.C. October 29,
More informationMaximizing take up to the Labour Force Survey with an initial online mode invitation
Maximizing take up to the Labour Force Survey with an initial online mode invitation Vicky Palmer & Andrew Phelps ONS Social Survey Division Data Collection Transformation at ONS Moving survey data collection
More informationLotteries Yukon s 2013 Household Survey and Web Survey Summary of Results
Lotteries Yukon s 2013 Household Survey and Web Survey Summary of Results Objectives of the survey Profits from the sale of lottery tickets in Yukon are allocated to local art, sport, and recreation projects
More informationThe Costa Rica 2010 Enterprise Surveys Data Set
I. Introduction The Costa Rica 2010 Enterprise Surveys Data Set 1. This document provides additional information on the data collected in Costa Rica between July 2010 and June 2011as part of the Latin
More informationSurveys. SWAL Alexander Klimczak & Lukas Mecke
Surveys SWAL Alexander Klimczak & Lukas Mecke Motivation To find out peoples opinions and interests asking each of them is oftentimes not possible Idea : Ask only some and try to generalize Survey Agenda
More informationDemand for Micro-Savings Programs in Uruguay: Motivation and Resistance to join C3U
Demand for Micro-Savings Programs in Uruguay: Motivation and Resistance to join C3U Magdalena Ramada Sarasola Innovation & Research Multiplier and Social Trade Organization (STRO) for the Institute for
More informationENERGY INITIATIVE ENERGY EFFICIENCY PROGRAM Impact Evaluation of Prescriptive and Custom Lighting Installations. National Grid. Prepared by DNV GL
ENERGY INITIATIVE ENERGY EFFICIENCY PROGRAM Impact Evaluation of Prescriptive and Custom Lighting Installations National Grid Prepared by DNV GL Date: September 25, 2015 DNV GL www.dnvgl.com September,
More informationPractical Sampling for Impact Evaluations
L.Chioda (adopted by DIME, LCRCE) Practical Sampling for Impact Evaluations innovations & solutions in infrastructure, agriculture & environment April 23-27, 2012, Naivasha, Kenya Introduction Now that
More informationMixed-mode surveys a comparison of sequential and parallel designs
Mixed-mode surveys a comparison of sequential and parallel designs Elena von der Lippe Jens Hoebel Stephan Müters Cornelia Lange Robert Koch Institute, Germany Ljubljana, 15-19 July 2013 Elena von der
More informationModifications to the Job Openings and Labor Turnover Survey
Modifications to the Job Openings and Labor Turnover Survey Charlotte Oslund Bureau of Labor Statistics 2 Massachusetts Avenue, NE, Suite 4840 Washington, D.C. 20212 Abstract The Job Openings and Labor
More informationA Survey on Survey Statistics: What is done, can be done in Stata, and what s missing?
A Survey on Survey Statistics: What is done, can be done in Stata, and what s missing? Frauke Kreuter & Richard Valliant Joint Program in Survey Methodology University of Maryland, College Park fkreuter@survey.umd.edu
More informationQuality report On Labour Cost Survey 2012 Macedonia. Prepared by: Elena Peeska Dejan Peeski
Quality report On Labour Cost Survey 2012 Macedonia Prepared by: Elena Peeska Dejan Peeski December 2014 CONTENT 1. Relevance... 3 2. Accuracy... 3 2.1. Sampling errors... 5 2.2. Non-sampling errors...
More informationOptimal Contact Strategy in a Mail and Web Mixed Mode Survey
Optimal Contact Strategy in a Mail and Web Mixed Mode Survey Anders Holmberg 1, Boris Lorenc 1, Peter Werner 1 1 Statistics Sweden, e-mail: name.surname@scb.se Abstract Statistics producers are increasingly
More informationUse of a nonresponse follow-up survey to assess the validity of R-indicators as a measure of the risk of bias
Use of a nonresponse follow-up survey to assess the validity of R-indicators as a measure of the risk of bias Caroline Vandenplas 1, Caroline Roberts 1 and Michèle Ernst Staehli 2 1 University of Lausanne,
More informationTacTex-05: A Champion Supply Chain Management Agent
In Proceedings of the Twenty-First National Conference on Artificial Intelligence (AAAI 06), Boston, MA, July 2006. TacTex-05: A Champion Supply Chain Management Agent David Pardoe and Peter Stone Department
More informationEnsuring the data collection quality of enterprise and household surveys an overview and new ways
Ensuring the data collection quality of enterprise and household surveys an overview and new ways Group of experts on Business Registers, Paris 27-29 September 2017 Merja Kemppainen Data collection at
More informationFARM STRUCTURE SURVEY 1999/2000 NATIONAL METHODOLOGICAL REPORT. Member State: DENMARK
FARM STRUCTURE SURVEY 1999/2000 NATIONAL METHODOLOGICAL REPORT Member State: DENMARK FARM STRUCTURE SURVEY 1999/2000 NATIONAL METHODOGICAL REPORT DENMARK TABLE OF CONTENTS SUMMARY...3 1. INTRODUCTION...4
More information