FACT SHEET ON QUANTITATIVE SURVEY METHODOLOGIES APRIL 2011 In contrast to qualitative research, which involves talking in- depth with a select number of people to find out why people think and behave the way they do, quantitative research involves talking to many people to quantify and validate insights, attitudes, facts and opinions. Quantitative research provides an understanding into how many and who holds particular views. Survey methodologies used to conduct primary quantitative research among the general population (as opposed to observational and data collection techniques, such as grocery scan) include: Face- to- face surveys Postal surveys Computer Assisted Telephone Interview (CATI) surveys Online surveys Automated telephone surveys. Face- to- face surveys and postal surveys were the first methods used for conducting quantitative research in Australia. From there, telephone interviewing became more commonplace over the years. More recently, online surveys have become frequently used and have all but removed the face- to- face methodology from the research landscape. New methodologies appear on a regular basis as technology progresses. Most recently, automated telephone surveying has become available. All of the above are proven and valid methodologies and the choice between them should be determined by the nature, objectives, required outcomes and budget of each individual research project. For the purposes of general population surveying, all methodologies should utilise probability sampling, which utilises some form of random selection to identify respondents for inclusion in the survey. Options include random sampling, systematic sampling, and stratified sampling. Probability sampling ensures that as far as is reasonably possible every person in your target population has a chance of being selected in the sample and this probability can be accurately determined. The combination of these traits makes it possible to produce unbiased estimates of population totals, by weighting sampled units according to their probability of selection. It also allows sampling error to be calculated. Page 1
Sampling error is the degree to which a sample might differ from the population. When inferring to the population, results are reported plus or minus the sampling error. Following is a description of each of the five survey methodologies listed above to conduct primary quantitative research and considerations for deciding when to use each. Face-to-face surveys Face- to- face surveying is less frequently used in commercial research than other quantitative methodologies due to the cost and time required to achieve similar geographic representation. The main advantage of face- to- face or direct interviews is that the interviewer can adapt the questions as necessary, clarify doubt and ensure that the responses are properly understood, by repeating or rephrasing the questions. The interviewer can also pick up non- verbal cues from the respondent. However, the cost for such interviews is relatively high. Often used to request quick feedback at the point of purchase or brand interaction, face- to- face surveys are generally most suitable for targeted and short surveys. Postal surveys Involving self- completion, postal surveys were cost effective pre- internet and a viable way to reach a larger number of people. The completed questionnaires can be automatically scanned for fast and accurate data turnaround. Postal surveys are less frequently used today for a number of reasons including the introduction of web interviewing as a viable alternative, relatively low response rates and they are prone to sample bias because they rely on respondent self- selection. These days postal surveys are most appropriate where simple visual stimuli are required to be shown, a large sample size is required and when respondents postal addresses are available and email address are not. Computer Assisted Telephone Interviews (CATI) The Computer Assisted Telephone Interview (CATI) survey method has been one of the most widely used methodologies for quantitative research in Australia for more than a decade. Page 2
CATI interviewing increases the speed and accuracy of carrying out telephone interviews by controlling the sample, quotas and questionnaire flow. With interviewers reading through a scripted questionnaire to respondents over the phone and entering the responses directly into the computer, it enables automatic routing and navigation depending upon respondents answers. It also assists in managing quotas, with multiple language projects and in interviewer supervision. Other benefits include immediate feedback of data and shorter field times than postal and face- to- face methodologies. CATI interviewing is particularly suitable for questionnaires up to 20 minutes in length that are highly structured or where questionnaires involve complicated routing. Occasions when CATI interviewing is less suitable are when you may have visual stimulus to prompt respondents with or if your target audience is skewed to demographic groups less likely to answer the telephone. Online surveys Online surveys have become commonplace with high online access rates by diverse segments of the Australian population enabling coverage across the market. Online surveys are typically sent via an email invitation either to a client- specific database or a research panel that people sign up to in exchange for incentives. In the case of online surveys utilising research panels, current levels of available respondents on Australian research panels make them suitable for surveying wider geographic locations such as nationwide, statewide or major capital cities. Representative samples size are harder to achieve at a local/regional or specific postcode level, for example. Research panel participants are also more likely to more frequently participate in research surveys than respondents reached through other methodologies, because they actively sign up to participate. Online surveying can be cost effective, providing high turnover rates, fast setup, execution and processing. Like the CATI methodology, online surveys also enable automatic routing depending upon respondents answers. Other significant benefits include the ability to show respondents visual stimuli and high response rates from allowing respondents to complete the survey at a time convenient to them, or even come back to it if they get interrupted. Online surveys are less suitable when targeting groups who are likely to be less computer savvy. Rather, online surveys are most suitable if your target audience is regularly online and if you have a quality list of respondents with accurate email addresses. They are also best if you have a relatively structured questionnaire format taking no longer than 20 minutes to complete. Page 3
Automated telephone surveys Among the newest additions to quantitative methodologies, automated telephone or Interactive Voice Response (IVR) surveys involve pre- recording a questionnaire and delivering it by automated calling. IVR surveys are only suitable for short questionnaires typically taking no longer than 5 minutes to complete and using closed- ended, simple, single response questions. The benefit of IVR surveys is that the automation of the telephone interviewing enables larger sample sizes to be achieved very quickly and more cost effectively than all of the above described methodologies. This makes the methodology ideal for reaching a large number of people with a limited number of questions. It can also be used for screening a wide population to identify a particular demographic or attitudinal subgroup, sometimes even as a pre- screener to then automatically be connected to a CATI survey interviewer. IVR also enables the cost- effective achievement of methodologically valid sample sizes in local and regional areas. JWS Research and quantitative surveying JWS Research frequently uses CATI, online and IVR quantitative methodologies, according to the requirements of each research project. We pioneered the introduction of IVR surveys to Australian public opinion polling at the 2010 Federal and Victorian elections. Our poll results have been quoted extensively in the Sydney Morning Herald, The Herald-Sun, News Online and in Channel 10 election coverage, and the methodology has been widely acknowledged by media, commentators and key decision makers. Our election polling shows IVR achieves comparable accuracy to other quantitative methodologies including CATI and online surveying used by other major pollsters. We are now applying this cost effective research technique for our commercial clients with great success. The IVR supplier used exclusively by JWS Research is a wholly Australian- owned company with Australian based servers and telephone lines. The validity of all the various quantitative methodologies lies in the numbers; a comparison of various methodologies is outlined below using published pre- election polling. Page 4
Tables 1 and 2 show a comparison of the published polling results for the 2010 Federal Election (Table 1) and the 2010 Victorian State Election (Table 2) conducted by the major pollsters, including methodologies and the difference between predicted outcomes and the actual election results. Both tables show the JWS Research pre- election IVR polls to be credible against other pre- election published polls, even without accounting for the time lag from the JWS Research IVR polls to each election. This analysis is purely for the purposes of highlighting the comparative accuracy of the IVR methodology. Identification of the most accurate pollster would require additional analysis, including analysis of primary vote results. Table 1: 2010 Federal Election Poll Accuracy JWS Pollster Actual Research Newspoll Nielsen Galaxy Morgan Essential Date 21 Aug 14-15 Aug* 17-19 Aug 17-19 Aug 17-18 Aug 21 Aug Aggregation of 2 weeks, released 16 Aug Methodology - IVR CATI CATI CATI CATI Online Sample size - n=28,000 n=2,507 n=2,040 n=1,200 n=1,220 n=2,160 ALP 2PP 50.1% 51.6% 50.2% 52.0% 52.0% 51.5% 51.0% LNP 2PP 49.9% 48.4% 49.8% 48.0% 48.0% 48.5% 49.0% 2PP Error - 1.5% 0.1% 1.9% 1.9% 1.4% 0.9% * Final JWS Research election poll was conducted one week before the actual election on 21 st August 2010. Other polling evidence shows there was a swing against Labor of approximately 1.8% on a two party preferred basis in the last week of the Federal election campaign, meaning the JWS Research poll on 14-15 August was likely very accurate against actual voting intentions at that time. Table 2: 2010 Victorian State Election Poll Accuracy JWS Pollster Actual Research Newspoll Nielsen Galaxy Morgan Date 27 Nov** 20-22 Nov* 23-25 Nov 24-25 Nov 23-24 Nov 22-25 Nov Methodology - IVR CATI CATI CATI CATI Sample size - n=9,218 n=1,451 n=1,533 n=800 n=990 ALP 2PP 48.4% 50.1% 48.9% 48.0% 50.0% 49.0% LNP 2PP 51.6% 49.9% 51.1% 52.0% 50.0% 51.0% 2PP Error - 1.7% 0.5% - 0.4% 1.6% 0.6% * Final JWS Research election poll was conducted one week before the actual election on 27 th November 2010. Other polling evidence shows there was a swing against Labor on a two party preferred basis in the last week of the Victorian election campaign, meaning JWS Research poll on 20-22 November was likely very accurate against actual voting intentions at that time. ** Most up-to-date figures available at time of writing. For further information about quantitative research or to discuss which methodology is best suited to your research project, please contact John Scales, Head of Research and Strategy at JWS Research at jscales@jwsresearch.com or on 0409 244 412. Page 5