UNDERSTANDING MOBILE RESPONDENTS VS DESKTOP RESPONDENTS The Need for Device Agnostic Surveys Position Paper: March 2017
The increase in worldwide smartphone usage is a widely discussed topic in market research, as companies struggle to reach consumers in an increasingly mobile world. Younger demographics are particularly difficult to reach, as they are more likely not only to prefer to take surveys on mobile devices but even to exclusively take a survey on their smartphone. As a result, surveys limited only to respondents completing them on a laptop or desktop may not be as representative of the population as device agnostic surveys or surveys that appear on all devices in an optimal way. 40% 30% 20% US Mobile Starts By Age/Gender 2015 Females Males Average 21% In Toluna s US panel, 21% of survey starts are currently from a mobile device, and 45% of panel recruits start from a mobile device. 10% 0% 14-17 18-24 25-34 35-44 45-54 55-54 65+ 2
Despite the evidence that mobile devices are increasingly utilized by respondents, demonstrated by the table opposite, mobileoptimized surveys suffer from commonly practiced industry limitations, as many surveys are not mobile optimized and are too lengthy and, or are not formatted properly. We conducted a study completed by respondents on multiple types of devices to examine 3 questions. 1. Does data differ based on the device on which the survey was completed? 2. Are respondents who utilize mobile devices to take surveys inherently different than non-mobile respondents in behavior and demographics? Country % Completes from Mobile Country % Completes from Mobile Country % Completes from Mobile Malaysia 30.5% Finland 24% Germany 17.2% South Africa 30.4% Canada 23.3% Spain 15.5% Ireland 29.7% Turkey 22.2% South Korea 14.2% Denmark 28.3% United States 22.2% India 14% Singapore 28.3% United Kingdom 21.7% Mexico 13.6% Norway 27.1% Australia 20.5% Brazil 12.7% Thailand 26.4% Switzerland 20.4% France 12.7% Indonesia 25.9% Japan 19.5% Russia 11.3% Austria 25.2% Portugal 18.5% China 7.7% New Zealand 24.8% Belgium 18.4% Greece 7.5% Hong Knog 24.5% Italy 18.2% Poland 4.7% Sweden 24.5% Netherlands 18% Czech Republic 3.8% % Completes from Mobile by Country Source: Toluna Respondents Chile 24.1% Argentina 17.8% 3. Does offering mobile compatibility and optimization increases the representativeness of the sample? 3
Background and Methodology In the study, we invited respondents to complete a 20-minute survey with eight lengthy grids on either their desktop/laptop or their mobile device. All respondents were sent multiple links to the same survey: a desktop/laptop link and a mobile optimized link. Respondents were then able to take the survey on their device of choice. This ability for respondents to choose their preferred device was intentional in an effort to increase our understanding of which respondents prefer mobile devices over desktops/laptops. We found that younger respondents, particularly younger females, were more likely to use a mobile device to answer their survey while older males were more likely to use a desktop. Respondents with a higher personal income were also more likely to take the survey from a desktop/laptop. After performing a driver analysis, we identified the main factors affecting device selection to be household income and age. We found that interview duration was similar for respondents regardless of device, although slightly more time was required for mobile respondents than desktop/laptop respondents. Respondents on a mobile device took an average of 23 minutes to complete the survey while desktop/laptop respondents took an average of 18 minutes. This was driven by mobile respondents needing more time to respond to more complex questions. Dropout behavior was also similar across devices. 60% of dropouts from the desktop link occurred within the first 2-3 minutes, while the same percentage dropped out of the mobile link in the first 6-7 minutes. Fraud levels were also not significantly higher for mobile respondents compared to desktop/laptop respondents. In fact, mobile respondents were less likely to be identified as straightliners 1 in the grid questions than those taking the survey on the desktop/ laptop. 1 Straightliners are respondents that choose the same answer for an entire grid of attributes or statements. 4
Data Comparison In terms of responses, we found a difference driven by behavior of respondents rather than device, emphasizing the importance of including mobile respondents to increase representativeness. Mobile respondents were more diverse in their answers to behavioral and attitudinal questions. Mobile respondents are more involved in various activities on their mobile devices as opposed to their desktop/ laptop. They also self-identify as concerned about more issues than desktop/laptop respondents, and they use fewer statements to describe themselves than desktop/laptop respondents. 35% 30% 25% 20% 15% 10% 5% 0% Several Noted Behavioral Differences Mobile vs Desktop Animal Welfare Bullying 40% 35% 30% 25% 20% 15% 10% 5% 0% Climate Change Education At home only 2 Food Safety / Food Location of Time Spent on Internet 3 Equally at home and out 5 6 Out of home only Issues That Concern Respondents Gender Inequality Gun Control Healthcare Immigration LGBT Issues Mental Health National Safety Racial Inequality Unemployment Wealth Inequality Other None of these Desktop/Laptop Mobile 5
Several Noted Behavioral Differences Mobile vs Desktop cont d Does This Statement Describe You? Desktop/Laptop Mobile It is important to this person to think of new ideas and be creative to do things his/her own way It is important to this person to be rich to have a lot of money and expensive things It is important to this person to be rich to have a lot of money and expensive things 56% 53% 23% 12% 59% 53% It is important to this person to have a good time to spoil him/herself 37% 32% It is important to this person to do something for the good of society 56% 56% Being very successful is important to this person to have people recognize his/her achievements Adventure and taking risks are important to this person to have an exciting life It is important to this person to always behave properly to avoid doing anything people would say is wrong Looking after the environment is important to this person to care for nature and save life resources Tradition is important to this person to follow the customs handed down by his/her religion or family 35% 30% 36% 33% 51% 44% 52% 47% 48% 41% The differences in behavior are clear and demonstrate the importance of device agnostic surveys for increasing the representativeness of the sample. 6
Key Takeaways Surveys limited only to desktops/laptops are likely to suffer in data quality due to the diminished capacity to reach certain segments of the population, particularly younger respondents and female respondents. Some of the other differences in behavior for mobile users compared to laptop/desktop users include more likely to notice advertising and more likely to take action after viewing advertising. Finally, they are more likely to spend time on the internet outside of their homes. Providing these respondents the ability to take surveys on their mobile device not only enhances the respondent experience, but it increases the likelihood they will start and complete the survey by making it available to them via their preferred device whether at home or another location. Satisfaction rates are similar across devices, with 83% top 2 box satisfaction scores for both mobile and desktop/laptop respondents. Furthermore, 2/3 of mobile respondents found taking the survey on a mobile device easier than on a desktop/laptop, and an additional quarter found it just as easy. Only 9% said it was difficult or much more difficult than on a desktop/laptop. Perhaps most interestingly, over half of mobile respondents were at least likely to take a 60-minute survey on their mobile device when asked about the possibility. However, over half of desktop/ laptop respondents were at least unlikely to do so. After being asked about interest in taking a 60-minute survey on a mobile device, respondents were then asked the same question with the additional ability to start/stop the survey and come back to it at a later time. When offered this option, the percentage of mobile respondents likely to take the survey on their mobile device jumped to 75%, and half of desktop/laptop users were now also likely to utilize their mobile device in that scenario. This is an important indicator of drawbacks on mobile devices, and we are conducting additional research on whether this feature increases the number of respondents that complete surveys on mobile devices and whether it will decrease the number of dropouts. 7
Device-Agnostic Recommendations and Limitations Overall, respondents on mobile devices displayed similar behavior in taking the survey itself in terms of dropouts and fraud, although differences in respondents across devices were clear. Younger, female respondents were more likely to use their mobile device to take the survey, and these respondents are likely to have differences in opinions and attitudinal behavior. However, this behavior is not driven by the device itself, but rather by the segment of the population more likely to complete the survey on their mobile device. When considering these differences in demographics, it emphasizes the importance of device agnostic surveys in order to increase the representativeness of samples for market research studies. Furthermore, mobile respondents reported high levels of satisfaction with a relatively long study of 23 minutes that included large grids, suggesting that length of the survey is less a barrier than previously expected. Additionally, 75% of respondents that completed the survey on their mobile device and half of respondents that chose to take the survey on their desktop/ laptop were likely to take a 60-minute survey on their mobile if given the opportunity to start and stop the survey. 8
Conclusions From this study, we conclude that differences in responses across different types of devices are not driven by the type of device used. Some traditional market researchers believe respondents taking a survey on their mobile device will not be able to provide responses that are as thoughtful or insightful as those taking the survey on their laptop/desktop, and some even believe mobile respondents are likely to be more fraudulent. However, we find the opposite results in this study. Furthermore, we find that allowing respondents to take surveys on their mobile device increases the representativeness of populations that are more difficult to reach, particularly younger respondents. As this group was more likely to complete the survey on their mobile device, it is important to allow for mobile completes when possible, since this will enhance the representativeness of the sample and improve data quality. It is important to consider that not all surveys are suited for mobile completes, although certain adjustments can be made in some surveys to make them mobile compatible without compromising data quality. Some best practices for device agnostic surveys include: Only asking absolutely necessary questions Reducing the amount of text in questions and prompts Simplifying survey logic when possible Limiting the number of open-ended questions Avoiding complex, tedious questions Avoiding complex grids Avoiding images that are too detailed or videos, as these may rely on mobile data and not load properly for respondents Avoid scrolling, particularly horizontal scrolling Make survey templates engaging and fun to answer on a small screen 9
While mobile devices are here to stay and respondents continue to complete surveys on them whether mobile-optimized or not, our research shows that allowing for mobile respondents can actually increase the representativeness of the sample. Younger populations more likely to utilize mobile devices have differing opinions than their older counterparts more likely to use a laptop/desktop, and these respondents are important to include in the sample. Furthermore, we find that mobile respondents are less likely to be fraudulent or speed through a survey, contrary to the beliefs of many market researchers. By allowing for device agnostic surveys and ensuring the guidelines for device agnostic surveys are followed, companies can better reach consumers and be more confident in their findings by ensuring their sample is the most representative it can be. 10