Examples of cost calculation for various mixed mode approaches: Cost drivers and their effectiveness Stefan Klug and Birgit Schrödle DemoSCOPE Research & Marketing AG, Switzerland 1
Mixed mode in (commercial) practise Mixing modes is a rare case where something cheaper can be better than something more expensive. It is not a quality against price trade off. But, cost reduction is the side-effect and not the primary purpose for doing mixed mode. The primary purpose is to achieve a higher response rate by leaving the choice of interview mode to the respondent. Therefore, the trade off is between a (higher) mode error against a (lower ) non-response error. We find that in practice mixed mode surveying was abused as a sales argument to shift interview mode from CATI to CAWI instead of selling the advantage of combined samples. I believe that by introducing additional mode effect variance the total estimate is closer to reality than the single mode estimate. If this does not hold true, it is still true that mode effects can at least be described and understood (instead of being ignored). 2
Mixed mode in (commercial) practise Mixed mode designs to study mode effect: An interpretation of mixed mode? Fixing the distribution of Web and CATI e.g. to 2 000 CATI and 2 000 CAWI. The purpose for this is to achieve equal sample sizes to facilitate comparison and to decrease costs on a fixed base (by shifting the commercial risk to the institute). An even more technical variant is to randomly assign and enforce the interviewing mode instead of leaving the choice to the interviewee. Purpose is the elimination of self-selection bias from the mode bias. Mixed mode without free choice of mode is an interpretation of mixed mode. Such designs are applicable if the understanding of the mode effect is more important than the reduction of the nonresponse bias (with free choice of mode). 3
Definition: Mode vs. Device Mode Device Drivers of mode effects (examples) Presence of interviewer Presentation of stimuli CATI CAPI Landline Phone Mobile Phone Portable PC (Laptop / Netbook) Tablet Computer Interviewer guided control of order and pace Acoustic Presentation / higher cognitive burden / help from interviewer available etc. Mobile (Smart) Phone Desktop PC or portable PC Visual Presentation CAWI Tablet Computer Mobile (Smart) Phone Self administered control of order and pace PAPI Paper 4
«Views on the MR world» There is no «CAMI» (Computer Aided Mobile Interviewing). CATI Landline + CATI Mobile = Dual Frame CATI There is only one internet: «CAWI Desktop» + «CAWI Mobile» = (still) CAWI Mixed Device (in CAWI) is a must, not an option Resarch in Social Networks is not a new mode but another sample recruitment source 5
Traditional Single Mode (zero node) Contact Mode Zero node Answer Mode Start No switching node. One contact mode assigned to each answer mode Answer Mode End Telefone CATI CATI Face to face CAPI CAPI Internet CAWI CAWI Letter of Invitation PAPI PAPI 6
Single Node Mixed Mode Contact Mode Single node Answer Mode Start Mode switching at a single defined time in contactinterview flow Answer Mode End Telefone CATI CATI Face to face CAPI CAPI Internet CAWI CAWI Letter of Invitation PAPI PAPI Single node can be placed anywhere but it has a fixed position in interview. 7
Multi Node Mixed Mode Contact Mode Single node Answer Mode Start Multi node Answer Mode End Telefone CATI CATI Face to face CAPI CAPI Internet CAWI CAWI Letter of Invitation PAPI PAPI Mode switching at multiple times in interview flow 8
Classification of Research Methodes Mode Nodes Device Single Mode Zero Node Single Multiple Mixed Mode Single Node Single Multiple Multi Node Single Costs for survey preparation increase Participation rate increases Completion rate increases Multiple Enforced mode vs. free choice as an additional classification dimension 9
Is it (financially) worth mixing modes? Yes, when the extra costs for mixed mode can be compensated through mixing more expensive with cheaper interview modes If the number of interviews from break-even analysis is achievable, than mixed mode has a positive cost effect If the extra costs for mixed mode setup are 0 than mixed mode offerings will always be cheaper than doing the more expensive mode For consequent mixed mode offerings a software platform is needed that reduces extra costs. But, mixed mode offering is not about cost saving it is about decreasing the non-response error (at the cost of introducing mode effects). 10
Mixed mode break even calculation 2 estimates + 1 simple calculation How much higher are the (fixed) costs if doing mixed mode compared to single mode? What is the price difference (variable costs) per interview between the most expensive mode and the cheapest mode? How many cheap interviews do I need to compensate for additional fixed costs through mixed mode? Value 1: Additional (fixed) costs for mixed mode = $20 000 Value 2: (variable) costs per interview expensive ($40) - (variable) costs per interview cheap ($10) = $30 Calculation: Value 1 $20 000 / Value 2 $30 = # Interviews 667 11
Example of mixed mode calculation Single Mode Mixed Mode COSTS CATI CAPI PAPI CAWI Best case Worst Case h $ h $ h $ h $ $ $ Setup $ 2'500 $ 2'500 $ 2'500 $ 2'500 $ 2'500 $ 10'000 Labor Costs Programming 10 $ 1'000 10 $ 1'000 10 $ 1'000 10 $ 1'000 $ 1'000 $ 4'000 Data base design/sampling etc. 10 $ 1'000 10 $ 1'000 10 $ 1'000 10 $ 1'000 $ 1'000 $ 4'000 Question wording 5 $ 500 5 $ 500 5 $ 500 5 $ 500 $ 500 $ 2'000 Contact $ 10'000 $ - $ 10'000 $ 10'000 $ 10'000 $ 10'000 Labor costs 200 $ 10'000 Adress Costs $ 10'000 $ 10'000 Interview $ 40'000 $ 70'000 $ 5'000 $ 100 $ 20'050 $ 20'050 Labor costs 667 $ 33'333 1000 $ 50'000 Other costs Telephone expenses $ 6'667 Travel expenses $ 20'000 Incentives $ 5'000 $ 100 Analysis $ 2'000 $ 2'000 $ 2'000 $ 2'000 $ 2'000 $ 8'000 Labor Costs Reporting 20 $ 2'000 20 $ 2'000 20 $ 2'000 20 $ 2'000 $ 2'000 $ 8'000 Total Costs $ 54'500 $ 74'500 $ 19'500 $ 14'600 $ 34'550 $ 48'050 Fixed Costs $ 4'500 $ 4'500 $ 4'500 $ 4'500 $ 4'500 $ 18'000 Variable Costs $ 50'000 $ 70'000 $ 15'000 $ 10'100 $ 30'050 $ 30'050 12
Costs and mode combinations $130 $120 $110 $100 CAPI only $90 $80 $70 $60 Equal mix CATI only CATI 75%, CAWI 25% $50 PAPI only $40 CAWI only $30 $20 $10 $0 Multiple contact, CAPI >=75% Multiple contact, CAPI + CATI >=75% Single or multiple contact, CAPI = 50% Single contact, CAPI >= 75% Multiple contact, CATI >=75% Multiple contact, CATI >=50% Multiple contact, CAPI and PAPI 50% Single contact, CATI und CAPI =100% Multiple contact, CAPI 50% and PAPI 25% Multiple contact, CAPI 50% and CAWI >= 25% Multiple contact, CATI und PAPI >=75% Multiple contact, CATI 75% und CAWI 25% Single or multiple contact, CAPI <=25% Single contact, CAPI =50%, CATI = 0% Multiple contact PAPI and CAWI >=50% Siingle contact, CAPI <=25% Single and multiple contact PAPI >=75% Single and multiple contact, PAPI>=75% Single and multiple contact CAWI >=75% Single Contact PAPI >= 75% Single contact, CAWI >=50% Total Costs per Interview Fixed Costs per Interview Variable Costs per Interview Contact costs per Interview Interview costs per Interview 13
Cost calculation scheme Fixed costs Fixed costs (basic) Fixed costs mixed mode Variable costs Variable costs contact (per interview) Variable costs interview (per interview) $100 Total Mode A Mode B Mode C Mode D Total CATI CAPI PAPI CAWI $ 18'000.0 $ 4'500.0 $ 4'500.0 $ 4'500.0 $ 4'500.0 $ 4'500.0 $ 4'500.0 $ 13'500.0 $ 4'500.0 $ 4'500.0 $ 4'500.0 $ 48'775.0 $ 12'500.0 $ 17'500.0 $ 3'750.0 $ 2'525.0 $ 20'000.0 $ 10.0 $ - $ 10.0 $ 10.0 $ 28'775.0 $ 40.0 $ 70.0 $ 5.0 $ 0.1 Number of interviews Share of contacts 200% 100% 0% 100% 0% Share of interviews 100% 25% 25% 25% 25% Number of interviews 1'000 250 250 250 250 Total costs $ 66'775.0 $ 17'000.0 $ 22'000.0 $ 8'250.0 $ 7'025.0 $120 Total costs per interviews $ 66.8 $80 $60 $40 $20 0% 25% 50% 75% 100% 14 $- CAPI CATI PAPI CAWI
Inbound CATI in mixed mode surveys Inbound CATI is a logical extension of the traditional CATI mode to decrease non-response leaving the free choice to call the institute at the best time. It requires: Invitation letter showing a (unique and toll-free) number Full daytime coverage in the call center Messaging and IVR routings Identification of caller in the questionnaire (e.g. printed ID on invitation letter) 15
SMS Invitation SMS invitation can be used to inform users of RDD mobile numbers about a study. Similar to an invitation letter sent to a landline phone related address. It requires: Short message text. (SMS costs are little but with high quantities they become expensive). Full multi-language text on a webpage. Response channels to opt out : e-mail, SMS, telephone We found no better response rates in a test on 2 000 RDD interviews. 60% 50% 40% 30% 20% 10% 0% No connection No ability Not in target group No willingness No availability Appointment Success SMS send No SMS send 16
Conclusion Cost saving is not the primary argument to do mixed mode (if you want to save costs: do CAWI) Mixing mode design should be extended to the full mixed mode approach leaving the choice of the mode to the interviewee and allowing for multiple switching nodes Mixed devices is a Must more than an option due to the interviewee s usage of these devices CAPI and PAPI should not be (generally) excluded from mixed mode designs It is recommended to use multiple contact modes Contact modes can be extended by Inbound CATI. SMS invitation can be useful (but not proven yet) 17