An Embedded Experiment to Test Non-Response Follow-Up Strategies when using Electronic Questionnaires

Transcription

1 WP. 27 ENGLISH ONLY UNITED NATIONS ECONOMIC COMMISSION FOR EUROPE CONFERENCE OF EUROPEAN STATISTICIANS Work Session on Statistical Data Editing (Oslo, Norway, September 12) Topic (iv): Using metadata and paradata to analyse the efficiency of editing processes An Embedded Experiment to Test Non-Response Follow-Up Strategies when using Electronic Questionnaires ABSTRACT I. Introduction Jeannine Claveau and Claude Turmelle, Statistics Canada 1. In, Statistics Canada launched a new project aiming at implementing Electronic Questionnaires (EQ) as the principal collection mode for business surveys. In preparation for the widespread adoption of EQ, an experimental design was conducted on seven current EQ surveys to evaluate different Non-Response Follow-Up (NRFU) strategies. The purpose of the study was to provide information to establish a standard collection follow-up strategy for EQ surveys. In essence, the study quantified the impact of using a telephone contact in comparison to using e- mail reminders for following-up with businesses that had not completed their survey. 2. This paper borrows heavily from Karaganis et al. (11) and Leung et al. (12). II. Overview of the UES Collection 3. The Unified Enterprise Surveys (UES) program consists of close to annual business surveys which are integrated in terms of content, collection and data processing. In 11, webbased EQ were developed and used to collect data for seven UES surveys, i.e. six surveys of the services industries and the head office survey. Before 11, only one of these surveys used EQ as a collection mode. 4. The UES collection process is performed in three steps. The first phase of the annual collection cycle for UES surveys consists of a telephone pre-contact conducted for new enterprises selected in the sample to confirm their contact information as well as the activity codes based on the North American Industry Classification System (NAICS). In 11, for the seven surveys with EQ, telephone pre-contact was conducted not only to confirm existing information, but also to obtain addresses of the businesses. Businesses were advised that collection would be done via electronic questionnaires and were asked to supply addresses. The only businesses assigned to paper questionnaires were those who adamantly refused EQ or 1

2 the ones who were not reached during pre-contact. For units who previously respond to EQ collection, a heads-up was sent instead to advise that their responses would once again be collected through EQ. 5. The second phase of the collection consisted of mailing out either paper questionnaires or invitations to the sampled units. The final phase of collection for the UES surveys consists of receiving completed questionnaires, imaging paper questionnaires, uploading both imaged and electronic questionnaires into the central collection system (Blaise) for edit verification and following-up with outstanding respondents (NRFU) and respondents whose questionnaire failed editing in Blaise (FEFU). UES collection is explained in more detail in Leung et al. (12). III. Methodology of the Experimental Design A. Embedded Experimental Design 6. An embedded balanced factorial design was used for this experiment. In the case of the UES it would not be feasible, due to the cost, to set up an experiment separate from the survey to study different non-response follow-up strategies. Conducting an experiment within a survey is normally cheaper than running a separate study, and with the recent economic climate this is an important consideration for us. 7. Recent research literature demonstrates that experiments embedded in ongoing sample surveys are particularly appropriate to investigate the effects of alternative survey methodologies on response behaviour (Van den Brakel and Renssen, 1998 and 5; Van den Brakel and Van Berkel, 2). Data collected using the current survey approach can be used to produce estimates and the complete set of data can be used for testing treatment differences. 8. An experiment within a survey can be thought of as a variation of a two-phase survey design, as illustrated in Figure 1. Since an embedded experimental design was used, testing if the treatments are significantly different requires the use of methods that take into account both the experimental phase and the sampling phase. The Wald test adjusted for the sample design as presented in Van den Brakel and Renssen (5) was used. Figure 1: Illustration of an experiment embedded in a sample survey. 2

3 9. In this paper, hypotheses are tested while taking into account the sample design, the experimental design, and the weighting procedure of the survey applied during estimation. Hypothesis testing can be done by using a simplified design-based Wald statistic by inserting design-based estimators for the subsample means and a covariance matrix of the contrasts between these subsample means. The design-based estimators for the subsample means assumes that each subsample can be considered as a two-phase sample, where the first order inclusion probabilities of the first phase sample are obtained from the sample design and the conditional first order inclusion probabilities of the second phase sample are obtained from the experimental design. B. Treatments. The set up of the experiment is illustrated in Figure 2. Note that the split between units receiving a paper questionnaire and an electronic questionnaire was not performed by randomization. Instead, it was determined by the respondents. Therefore, we cannot compare the results between paper and EQ collection. In fact, paper collection was composed of businesses that were not contacted during pre-contact and those who refused to use EQ. Thus, these businesses are more prone to be a non-response and are suspected to be qualitatively different from the units who went with the EQ collection mode. Figure 2: The set up of the experiment. Overall sample Telephone pre-contact or heads-up Paper questionnaire mail-out on Mar 18 EQ mail-out on Mar 28 NRFU by fax reminders and/or phone 1 then + Phone follow-up started + 3 s, 4 s, one every 2 weeks Phone follow-up started + 3 s, one per month 1 phone attempt then + 3 s, one every 2 weeks one per month 3

4 11. Businesses that participated in EQ collection were randomly assigned to one of four different follow-up treatments (combinations of telephone and reminders at different points throughout the collection period). The various treatments differ by the frequency of their follow-up attempts and whether or not respondents were called. Each treatment contained approximately 25% of the EQ sample. The randomization was done within each combination of survey, strata group and type of questionnaire. The three strata groups are: (1) small take-some stratum, (2) large take-some stratum and (3) take-all and must-take stratum. The type of questionnaire was identified by its length, i.e., long form or short form. It has to be noted that stratification was applied to six of the surveys while the Head Office survey was a census. As a result, we ended up with a randomized block design for service industries (surveys and strata used as blocks) and a completely randomized design for the Head Office survey. 12. On March 28, 11, each unit in Treatments 1 to 4 ( to ) received an invitation that contained a hyperlink and an access code for completing the survey online. Nonresponse follow-up started approximately one month into the collection, on April 26, 11. On that date, all outstanding units were sent to NRFU by a specified approach indicated for to. Each treatment had a different NRFU approach using a combination of reminders and phone attempts. NRFU stopped for a unit as soon as the questionnaire was returned to Statistics Canada. 13. The four treatments were designed as follows. Treatment 1 was designed to replicate the standard NRFU strategy. We sent a first reminder to the outstanding units then phone follow-up started. Three others reminders were also sent once per month. Treatment 2 was designed to test how far we could get with reminders only, i.e. what kind of response rates could we achieve without any phone follow-ups. We did not allow telephone follow-up, but if an enterprise called the help line at Statistics Canada, interviewers would call them back as needed. Because there was no phone follow-up at all, we decided to send reminders more often, once every two weeks. Treatment 3 was designed to measure the impact of doing the first followup attempt by phone rather than by . In this case, phone follow-up started before sending e- mail reminders, once per month. Treatment 4 was designed to combine both having the first follow-up attempt done over the phone and then switching to only reminders every two weeks. 14. The embedded experiment was to be conducted on live surveys, with the collected data to be used to produce usual estimates. This placed a significant constraint on the experiment. We had to ensure that the results at the end of collection would only be minimally affected. Therefore, it was decided to stop Treatment 2 and 4 on June and then to switch to a final blitz of phone follow-up on all outstanding units. This was decided because the potential risk to damage the final response rates was assumed higher for Treatment 2 and 4 since these 2 treatments did not benefit from continuous phone follow-up from the beginning. The experiment ended on August 1, 11 for Treatments and. Officially after this date, all non-response units were switched to follow-up by telephone, regardless of their treatment. In reality, telephone follow-up for Treatments 2 and 4 began on July 8. 4

5 IV. Results A. Descriptive Statistics 15. There were a total of 9,324 units sampled for these seven surveys. Among these units, 6,457 (approximately % of the overall sample) were assigned to EQ collection after precontact. These EQ units were then divided randomly into four treatments of approximately the same size. The four treatments had 1,615, 1,613, 1,615 and 1,614 units respectively. 16. Through follow-up of non-response, respondents could request a switch of a collection mode. Thus during collection, we observed 338 units switching from paper to EQ and 521 units switching from EQ to paper collection. Also, it was determined during collection that 1,98 units were out of scope. At the end of collection, the numbers of in scope units in the four treatments were 1,375, 1,353, 1,394 and 1,376, respectively. B. Return Rate 17. The progression over time of the number of questionnaires returned (cumulative unweighted return rate) for each of the four follow-up treatments is shown in Graph 1. This unweighted return rate based on the number of questionnaires returned is useful to assess the performance of a collection process, independently of the business size. The return rates were computed based on all in scope units at the end of collection. 18. All the events that happened on the key dates are indicated by the dotted lines labelled from E1 to E5. The first event, E1, was the beginning of the non-response follow-up. This first event happened at the end of April. Treatments 1 and 2 received an reminder and Treatments 3 and 4 received a phone attempt during that week. At the second event, E2 at the beginning of May, businesses subject to Treatments 2 and 4 received an reminder. At the third event, E3, at the end of May, all four treatments received an reminder. Event E4, at the beginning of June, was the last reminder for Treatment 2 and 4 before the end of experiment for these 2 treatments. Finally, event E5 represents the start of regular telephone NRFU for Treatment 2 and We can observe on the graph that, at the beginning of the non-response follow-up, the number of questionnaires returned for the four treatments were similar (between 14 to 16%). Two weeks later (on May 9, event E2 Graph 1), all the treatments had similar unweighted return rates (between 23 to 25%). But, after Treatments 2 and 4 received their second reminder, we observed a substantial increase in response. The lead in response was maintained until the end of May for Treatment 2 and until mid June for Treatment 4. It seems that having a reminder action every two weeks instead of every month is beneficial in obtaining responses. However, the reminder (event E4) on June 7 had little impact on response. The effectiveness of only sending e- mail reminders clearly diminishes over time. In fact, by June, after three months of collection, the businesses that received telephone follow-up showed much greater gains. But from another point of view, the reminders only treatment (Treatment 2) managed to obtain its first 45% of response simply by sending reminders every two weeks. By the end of collection, all test groups had similar return rates. The return rates for all four treatments were over % at the end of collection in mid-october. 5

6 4/11 5/9 6/6 7/4 8/1 8/29 9/26 Graph 1: Cumulative unweighted return rate observed on each day during collection for each treatment (March 28 to October 14, 11) E1 E2 E3 E4 E5 E1 -> April 26 /: /: phone attempt E2 -> May 9 /: E3 -> May 26 ///: E4 -> June 7 /: E5 -> July 8 /: NRFU started. When we looked at the seven surveys separately (Graphs 2 to 8), we observed more variability among the treatments for the unweighted return rates. For some surveys, the return rates for the treatments using continuous phone follow-up (Treatments 1 and 3) were slightly higher than those of the other two treatments (Graphs 2 and 3). But for other surveys, the treatments with no phone follow-up at the beginning (Treatments 2 and 4) seemed to give the highest rates (Graphs 4 and 7). Although some differences in the return rates were observed among the surveys, by the end of collection, the return rate for each treatment in each survey was always higher or equal to 74%. In some cases the percentage was over % and it sometimes even reached %. 21. We also looked at the graph of cumulative revenue weighted return rate. This revenue weighted return rate takes into account both the sample weight and the importance of the units in terms of business size. The revenue variable used to weight the return rate is the revenue variable available on the Business Register of Statistics Canada. This graph is similar to the unweighted rate in Graph 1. 6

7 Graph 2: Head Office Graph 3: Specialized Design Head Office Specialized Design Graph 4: Computer Services Graph 5: Architecture Computer Services Architecture 7

8 Graph 6: Engineering Graph 7: Accounting Engineering Accounting Graph 8: Consulting Consulting 8

9 C. NRFU Costs 22. The total cost of collection for UES surveys includes pre-contact, mail-out of questionnaires, NRFU, FEFU, capture and imaging, management and other tasks such as helpline, quality control monitoring, system testing, etc. In this paper, given the focus on NRFU strategies, we will concentrate on NRFU costs defined through the number of NRFU attempts and the length of such attempts among the four treatments. 23. A NRFU phone attempt is recorded in the Blaise System when an interviewer tries to contact the respondent at the time when a questionnaire has not yet been received by Statistics Canada. The start and end times of each attempt are recorded along with the outcome of the attempt. 24. The percentage of in-scope units having at least one NRFU call attempt by the end of collection (October 14) for each treatment is presented in Graph 9. As seen from Graph 1, approximately 15% of the units in each treatment returned their questionnaire by April 26. Only the remaining outstanding units were eligible for NRFU. 25. Treatment 2 had the smallest proportion of units getting the NRFU attempt, because outstanding units in Treatment 2 received only reminders in the first three months of the experiment. Telephone follow-up did not start until July 8 for this treatment. Note that some units in Treatment 2 received a call during the experiment because it was agreed at the beginning that if an enterprise called the help line at Statistics Canada, interviewers would call them back as needed. 26. Treatments 3 and 4 had the highest number of units with at least one phone attempt due to the set up of the experiment: outstanding units in these two treatments received a phone attempt during the week of April 26. For Treatment 1, the outstanding units did not receive any telephone follow-up until after receiving the first reminder. Thus, the percentage of Treatment 1 units with phone attempts was slightly lower than for Treatments with the initial follow-up attempt done over the phone. Graph 9: Percentage of in scope units that received at least one NRFU call attempt by the end of collection, October 14, The average number of call attempts for each in scope unit in the four treatments as of June and October 14 are presented in Graph. Recall that a maximum of five NRFU phone attempts was allowed before a unit was considered as a final non-response unit. Recall also that not all units were eligible for NRFU (as they might have submitted their questionnaire by the time NRFU started) and even if they were, not all outstanding units were contacted by phone due to set up of the experiment. As of June 9

10 Number of attempts, Treatment 2 was the only treatment that received reminders only. It is therefore expected that Treatment 2 would have a very low average number of attempts. The average started to pick up after July 8 since all outstanding units in the four treatments received telephone follow-up. By the end of collection, the average number of attempts for Treatment 2 was still the lowest, while for the other three treatments, the averages were similar. As expected, Treatment 3 had the highest average number of NRFU attempts as it included both phone follow-up throughout the experiment as well as the initial phone follow-up attempt on April 26. It also appeared that the effect of doing follow-ups and doing the first attempt over the phone are similar, in terms of average number of NRFU call attempts, as illustrated by Treatments 1 and 4. Graph : Average number of NRFU call attempts for each in scope unit in each treatment (on June and on October 14, 11) Average number of NRFU call attempts for each in scope unit June October 14.3 Treatment 1 Treatment 2 Treatment 3 Treatment As mentioned before, the start and end times for each attempt are recorded in the Blaise System. We are then able to compute the duration of the attempts, also known as Time Per Unit (TPU). One of the goals of this experiment was to find the most efficient non-response follow-up strategy, i.e., the one producing the best return rate at the lowest cost. Based on descriptive statistics, Treatments 1 and 3 were the treatments with the most intensive follow-up: they received reminders and telephone follow-up from the end of April. Consequently, they were also the most expensive. Treatment 4 was less expensive than Treatments 1 and 3 because there was no telephone follow-up until July 8, but there was one phone attempt at the beginning of NRFU. The least expensive strategy was Treatment 2, in which only reminders were used at the beginning and there were no telephone attempts or follow-up until July 8. The scatter plots of the unweighted return rate versus the TPU for non-response follow-up on June and October 14 are shown in Graph 11. Since it is more costly to perform telephone follow-up than to send e- mail reminders (which is an automated process), a treatment was considered more costly if the TPU on NRFU was higher. Therefore, from the scatter plots, we observed that Treatment 2 was able to give similar return rates with lower cost on phone follow-up (around 11 minutes by the end of collection), compared to (around 15 minutes) for the other three treatments. The difference represents approximately 25% cost reduction compared to phone-only NRFU (excluding overhead cost).

11 Graph 11: Scatter plots of unweighted return rate versus the average duration of NRFU attempts (TPU) (on June and on October 14, 11) D. Analysis of Variance 29. Analyses of variance on return and response rates, number of NRFU attempts and time spent per unit (TPU) on non-response follow-up and failed edit follow-up were performed to compare the different treatments. Recall that the Head Office Survey is a census where each unit has the same sampling weight, while for Services surveys a stratified sample is selected. Because of the difference in the sampling designs, the analysis of variance tests were run separately for Head Office and for Services surveys. For tests on Head Office units, the F-Test in the SAS procedure PROC GLM was used. For Services surveys, the Wald Test adjusted for design (Van den Brakel and Renssen, 5) was used.. In Table 1, the results of the analysis of variance for Services surveys show that the unweighted return rate was significantly different between the four treatments on June. Unweighted response and revenue weighted rates were not significantly different on June. At the end of collection, there were no significant differences between the four treatments in terms of return and response rates. For the Head Office survey, unweighted return and response rate were not significantly different between treatments on both June and October 14 dates. 31. On the other hand, the number of telephone follow-up attempts and the TPU were found to be significantly different. When we looked at the seven surveys separately, the number of telephone followup attempts and TPU were always lower for the reminders only treatment (Treatment 2). 32. Analyses of variance on the number of NRFU attempts and time spent per unit (TPU) on failed edit follow-up were performed to compare the different treatments. For all surveys, neither variable was significantly different between treatments. 11

12 Table 1: Testing treatment differences on June and on October 14 Main Effects: = = = Head Office Jun Oct 14 p-value p-value Services Jun Oct 14 p-value p-value Revenue weighted return rate Revenue weighted response rate Unweighted return rate Unweighted response rate NRFU attempts < <.1 <.1 FEFU attempts NRFU TPU < <.1 <.1 FEFU TPU V. Conclusions 33. From a collection perspective, the design of experiment showed that significant response can be obtained without making phone calls for the first two or three months of collection for annual surveys. In addition, more frequent reminders are beneficial for obtaining responses. The strategy consisting of reminders every two weeks at the beginning of collection succeeded in obtaining the first 45% of response at lower cost. On the other hand, there is a point in time when it becomes necessary to start telephone follow-up, but starting this process later does not appear to negatively impact the final survey response rates. References Karaganis, M., Fox, K., Claveau, J., Leung, J. and Lin, W. (11). Embedded Experiment for Nonresponse Follow-up Methods of design of Electronic Questionnaire Collection. Proceedings of Statistics Canada Symposium 11. Leung. J., Claveau, J., Turmelle, C., Fox, K., Lin, W. and Karaganis, M. (12). Experimental Design for Non-response Follow-up of an Electronic Questionnaire Survey. Proceedings of 12 Federal Committee on Statistical Methodology Research Conference. Van den Brakel, J.A. and Renssen, R.H. (1998). Design and Analysis of Experiments Embedded in Sample Surveys. Journal of Official Statistics, 14(3), Van den Brakel, J.A. and Renssen, R.H. (5). Analysis of experiments embedded in complex sampling designs. Survey Methodology, 31(1), 23-. Van den Brakel, J.A. and Van Berkel, C.A.M. (2). A design-based analysis procedure for twotreatment experiments embedded in sample surveys. An application in the Dutch labor force survey. Journal of Official Statistics, 18(2),