White Paper USER TESTING 101: THE MARKETER S CRASH COURSE IN USABILITY TESTING 1
Usability testing is the direct observation of a user s real time interactions with a site, usually used to gather user experience data. With new advances in technology, a constant influx of distractions, and competition from more brands, your ability to exceed user expectations continues to grow in importance. There is convincing evidence that usability testing, when done correctly and interpreted accurately, has positive impacts on ROI, conversion rates, and customer loyalty. 110% ROI when you invest in usability testing Emerging technologies, and companies specializing in this task, make it easier and less expensive to do the minimum amount of testing needed for insightful observations. Usability testing allows you to: n Get direct and honest feedback from your user base 1 test with 1 user 38% Usability Improvement n Discover pain points that hinder your customers from reaching goals n Compare task success rates after you ve made design improvements n Test and validate new additions to the site 83% A site redesign based on usability User Experience Improvement n Understand if any deviations from standard web best practices are helping or hindering your audience Source: Nielsen Norman 1
When do you need to test? There are a few critical times when testing should be done. You absolutely should incorporate usability testing: n When releasing a new product or feature to the site n Before launching a redesign, and ideally at intervals during the process n If your analytics points to abandonment n To reinforce and compliment your customer centric strategy It s also not a bad idea to set up user testing at regular intervals. Setting up consistent usability testing keeps you responsive to your customers expectations, and continually removes obstacles in their path. It also gives you benchmarks and trends you can use to improve usability metrics. When conducting usability testing, it s important to remember that there are also internal users of the site. Your own stakeholders, the authors creating content for the site, and the web team responsible for development are all important contributors to the final product. 2
What questions do you need to answer to set up testing? Before you can determine what kind of testing best suits your needs, there are some important parameters that will help to define your actual methods and desired outcomes. 1. What exactly is being measured? Clearly define the critical tasks that will be tested. Knowing exactly what a customer s important goals are on your site, and quantifying their success in accomplishing them, is an important part of the plan. You can state both broad (can users identify important information on a page) and specific (can users locate a membership form) goals for the user. Each goal must have a measurable outcome that will determine if the user achieved it. Ideally, each test should not encompass more than a dozen tasks. This keeps the testing at a manageable level for the user, the moderator, and those analyzing the data. For mobile testing, each test should be limited to no more than eight tasks. Critical Tasks should: n Represent the most common user goals n Encompass the conversion goals of the site owner n Contain very clear success criteria n Define the starting point of the task 3
What questions do you need to answer to set up testing? - Continued 2. Who are your test participants? Participants can be representative users, or just randomly chosen individuals. Representative users match pyschographics and demographics of your actual target audience. Psychographics have to do with attitudinal insights and goals (personality traits, values, lifestyle, interests), whereas demographics are age, gender, device usage, etc. Deciding whether you are going to use representative users, or a random sampling, may be determined by time, budget, and what you need to measure. 3. What do you hope to learn? If you are testing an existing system, you will want to know the frustrations, blockers, and bugs users encounter, so you know what needs work, and your moderator can ask them what they d expect to see. If you re testing a proposed system, you want to learn what still needs work, what is/isn t working, or how much better particular task flows are working when compared with the old system. It takes 12 positive experiences to make up for 1 unresolved negative experience. Source: Understanding Customers by Ruby Newell-Legner 4
What are the different types of testing? While there are many different usability studies you can perform, most of them fall within five different categories of execution: 1 - Hallway 2 - Remote Unmoderated 3 - Remote Moderated 4 - Lab Moderated 5 - On-Site Moderated Your end goals, user-base complexity, budget, and timeframe will dictate which is best for you. Utilizing a combination of these, depending on where you are in the development cycle, is a sound strategy to take advantage of direct feedback, while staying efficient. The method of testing you choose will be based on several factors including: n Available time and resources n Cost n Task complexity n Testing frequency n Audience demographic requirements n The overall goals of the testing 5
1-Hallway: A quick, inexpensive method in which randomly selected people e.g. those passing by in the hallway are asked to try using the product or service. This can help designers identify brick walls, problems so serious that users simply cannot advance. It s best used in the early stages of a new design. 10 Tips for Better Hallway Testing: 1. Choose the right location 2. Plan ahead 3. Set up early 4. Review and practice 5. Get greeters 6. Be mindful of time 7. Explain the purpose to the participant 8. Reward your volunteer 9. Debrief after test 10. Look to improve Source: www.digitalgov.gov PROs: n Valuable perspectives you might have otherwise overlooked n How some users may actually behave, information that can help to test hypotheses n It s quick, easy, and very inexpensive n Provides outside, unbiased feedback n It can be performed almost anywhere and administered by most people CONs n It cannot show how your specific user personas use the site n There is much less confidence in the sampling, meaning more room for error n There is no formal documentation to refer to, i.e. audio or video n You can only test very basic concepts and ideas n It cannot complement nor help build accurate personas 6
2-Remote Unmoderated: It s a cost-effective method that uses more targeted demographics to choose testers. Companies like usertesting.com have made a great business giving clients access to willing participants, and the technology to complete the process. It s a solid starting point when exploring more focused areas of your site, but it lacks the capability to observe and conduct follow up. PROs: n Accurate users, often with hassle-free recruitment n Quick and inexpensive n It can be performed by users in their natural context, making collected data more accurate CONs n Testers may not actually be users of your site n Cannot entice follow up feedback or ask things like what would you have expected to happen when a road block occurs n Testing is limited to a truncated task list and set time duration n Facilitators can t offer clarity if task wording was difficult to understand 7
3-Remote Moderated: With a variety of technologies available, the ability to communicate and elicit feedback from users in remote locations is becoming easier to accomplish. The biggest advantage to remotely moderated testing is that you can often keep users in their natural environment, allowing you to duplicate the environmental and lifestyle factors that affect their online interactions. This is a also great method of testing when you have the need to cover a wide geographical area, but are dealing with constraints on time and budget. PROs: n User is tested in their natural environment n Geographical constraints are eliminated, allowing a larger testable audience n Moderator can probe the user for deeper insights when needed n Accurate user types What makes a good moderator? n Doesn t lead user or offer any bias n Accurately documents body language n Ability to decipher user s verbal and nonverbal feedback n Can entice the user to provide deeper insight n Understands the value in well-timed silence CONs n Visual cues from the user may be hard to interpret n Customized software is necessary n A very skilled moderator is necessary to remotely communicate and interpret cues n Technical difficulties are very common with this method 8
4-Lab Moderated: When testing in your own lab, you have the ability to pick up on even the most minor changes in body language, which can cue the moderator to encourage verbal feedback from the user. You have total control over the testing environment, eliminating any distractions and maximizing data collected. PROs: n Moderator can probe the user for deeper insights when needed n Visual cues/body language are easy to pick up n Accurate user types n Can tailor/personalize test based on user feedback A lab environment can give false results when contextual cues are not duplicated. For example, a parking app that conducts testing in a lab might see users put their phone on the table and use their pointer fingers to tap through the app. But in reality, the app is most often used while driving, with a thumb making selections and only peripheral vision. Examples like this should be considered when determining the best environment for generating real data. CONs n Can require a significant resource investment n User recruitment may prove difficult and costly, especially within differing geographical regions n Users are in an engineered setting important contextual data could be missing 9
5-On-Site Moderated: With this method, the moderator sits with users in their own environments to conduct the tests, giving users the highest level of comfort. This also allows the moderator access to the users environmental and lifestyle factors, which may directly impact their online experience. For example, you might see the user has sticky notes with reminders to site addresses, task paths, or passwords. This kind of testing gives you the most accurate data, yet is the hardest to coordinate. Remember that someone s natural environment is not always their cube at work. Someone might be on their couch at home, or on their laptop in a cafe. This is why context is so important. With mobile studies, people might be on the train or on their way to work, where distractions could be very common. Take the time to understand where your users are most likely to be when they are active on your site. PROs: n Contextual cues essential to usability are accounted for n Spontaneous tester feedback can be elicited with a skilled moderator n Qualitative and quantitative data generated is the most robust and accurate CONs n Coordination of user, environment, and moderator can be costly in time and dollars n The presence of a moderator in a user s personal space can create a level of discomfort n Because of the resources needed, it may not be possible to test multiple users in each demographic 10
Quick Reference Table 1 2 3 4 5 Hallway Remote Unmoderated Remote Moderated Lab Moderated On-Site Moderated Good For... A quick, high level perspective on blockers you may have overlooked Accurate users for a quick, focused task list Encouraging deeper feedback from a geographically diverse audience Controlled environment giving moderator ultimate control of testing to solicit in-depth feedback Uncovering environmental factors that play a key role in user experience Bad For... In-depth user experience analysis, actual site user feedback Evoking deeper feedback from user interactions Overcoming technical difficulties and interpreting user visual cues for timely feedback Capturing natural artifacts or visual cues from users, ensuring actual user behavior Navigating distractions of a real world setting 11
CONCLUSION Usability testing offers a variety of ways to get direct feedback from users, and better understand how they interact with your products and website. When set up and performed correctly, you will gain valuable insights that will help you to identify changes that can improve performance and satisfaction levels. As Jakob Neilsen said, A bad website is like a grumpy salesperson, and because oftentimes it is your only contact with customers, your site needs to exceed their needs and expectations. Usability testing gives you the knowledge to provide that great experience. A bad website is like a grumpy salesperson Jakob Neilsen Usability testing offers a variety of ways to get direct feedback from users, and better understand how they interact with your products and website. 12
USER EXPERIENCE ASSESSMENT Direct observation of a user s real time interactions with a site prove time and again to be invaluable in building stronger experiences. Usability testing has proven itself to directly impact ROI. It s simple really, happier customers lead to a better bottom line. Our experts design and conduct specific usability testing based on the needs of your customers, and tailored to meet the needs of your business. Using direct customer feedback and our analysis of the data, we design and architect intuitive, user-centered web sites for Adobe Experience Manager (AEM) and other platforms that provide author efficiency, deliver great customer experience, and move business KPIs. Interested in learning how usability testing can impact your business? CONTACT Mark Kelley mark.kelley@icidigital.com 919.883.9467 x124 Corporate Office: 4000 Westchase Blvd., Suite 280, Raleigh, NC 27607 Additional Locations: Chicago, IL Denver, CO Washington, DC icidigital delivers marketleading enterprise technology solutions that give you a decided advantage. We create a trusted partnership with you and your business to effectively plan, design, and build digital strategies, interactive experiences, and enterprise platform solutions. Our creative, business, and technology experts integrate seamlessly with your team to strategize how to best leverage collective talents, maximize efficiencies, and produce the exact outcomes you desire. icidigital s team of experienced specialists, plus our track record of successful partnerships, gives us an edge in industry know-how and an understanding of the big picture. With expertise in AEM, and experience building adaptive systems with Adobe Experience Cloud, icidigital is a powerful implementation partner for enterprise organizations. 13 13