Overseas usability testing

Transcription

1 Overseas usability testing A Case study BY JOE joedyer@gmail.com

2 A long time ago in a country far, far away... I had the remarkable experience of managing the usability tes;ng of a prototype in Europe across three markets.

3 by the numbers 12 days 10,723 miles 3 tes;ng loca;ons 1 prototype 3 languages 24 test par;cipants

4 by the numbers 12 days 10,723 miles 3 tes;ng loca;ons 1 prototype LONDON PARIS MILAN 3 languages 24 test par;cipants

5 Who is this for? usability experts

6 Who is this for? Business or product managers

7 Who is this for? the morbidly curious

8 Who is this for? wait, what are we talking about? the morbidly curious

9 is testing Different overseas?

10 How testing is different overseas INTERACTIVE AGENCY Tes;ng abroad almost always involves working with a tes;ng agency. Think expensive contracts.

11 How testing is different overseas Less than favorable exchange rates in the last several years

12 How testing is different overseas London milan Have you seen the cost of interna;onal travel these days?

13 How testing is different overseas He says, your breath smells like garlic! He says, your breath smells like garlic! votre haleine sent l'ail! Transla;on services are likely to be an addi;onal cost.

14 4

15 what worked well and what did not? preparation facilitation execution

16 Plan early & Share early Test objec;ves and scenarios were wriren internally before shared with tes;ng consultants.

17 appropriate # of tasks Number of test objec;ves in the usability test plan were too numerous for the test as it was executed.

18 appropriate # of tasks Facilitators wasted too much ;me on unrequested benchmark tes;ng ques;ons.

19 Engage early & often agency client Maybe two conference calls and a dozen or so s were not quite enough.

20 Engage early & often ZZZZ agency client Maybe two conference calls and a dozen or so s were not quite enough.

21 Test preparation Do all that you can to assess the skills and abili;es of any agency or contractor you intend to hire. Get mul;ple bids if you can. Over communicate all tes;ng objec;ves un;l they are capable of being your proxy.

22 the prototype We created a single prototype, in three languages.

23 go high-fidelity Limited sta;c HTML model made it nearly impossible for test par;cipants to explore the possibili;es of the prototype.

24 lost in translation Visitors are expected to complain at the office between the hours of 9 and 11 am daily Some translated words were too literal and were not reviewed for context within the prototype.

25 Please leave your values at the front desk

26 The genuine antics in your room come from our family castle. Long life to it.

27 special today: no ice cream

28 special today: no ice cream our wines leave you nothing to hope for

29 We take your bags and send them in all directions

30 We take your bags and send them in all directions The manager has personally passed all the water served here

31 Come alive with the Pepsi Generation

32 Pepsi will bring your ancestors back from the dead

33 Salem - Feeling Free

34 When smoking Salem, you feel so refreshed that your mind seems to be free and empty

35 selecting test participants The English call screener was easy to review and approve. The French screener? The Italian screener?

36 printed test scenarios Scenarios were given to par;cipants wriren on a card for consistency.

37 and even more issues... Facilitators were not familiar with the prototype or test objec;ves prior to tes;ng.

38 keeping it on the fairway

39 keeping it on the fairway How am i doing?

40 keeping it on the fairway How am i doing? do you really need me to tell you again?

41 test facilitation Instant Messenger can be your friend.

42 the domino effect

43 the domino effect Long process of contrac;ng, legal review, etc

44 the domino effect Long process of contrac;ng, legal review, etc Test plan covered in two conference calls &

45 the domino effect Long process of contrac;ng, legal review, etc Test plan covered in two conference calls & Benchmark tes;ng encroached on scenarios

46 the domino effect Long process of contrac;ng, legal review, etc Test plan covered in two conference calls & Benchmark tes;ng encroached on scenarios Facilitator s lack of familiarity with prototype

47 beware expensive reports draft report final report

48 so, why three cities at nearly the same time?

49 underscore user differences MARKET COMPARISON LARGE medium SMALL

50 underscore user differences MARKET COMPARISON LARGE medium SMALL i 1. Very internet savvy 2. High rate of adoption 3. Not likely to need offline help 4. market is ready for the solution

51 underscore user differences MARKET COMPARISON LARGE medium SMALL i 1. more market potential 2. not likely to need offline help 3. solution has a chance to succeed with some modifications

52 underscore user differences MARKET COMPARISON LARGE medium SMALL i 1. catastrophic usability findings 2. Lower rate of adoption 3. most likely to need offline help 4. solution doesn't work in this locale

53 reduced travel costs

54 reduced travel costs

55 so biz managers could see market reactions Key business managers were present during the tests, directly observing poten;al customers interac;ng with the new e- commerce solu;on.

56 the time machine

57 the time machine just getting to work

58 the time machine just getting to work last test of the day

59 the time machine just getting to work last test of the day where to eat dinner?

60 this is no time for a vacation All resources working on the prototypes should not be on vacation or tied up with other commitments during the time of testing. Consider moving to an alternate schedule.

61 technical difficulties! Sorry. The streaming video has timed out. See your System Administrator for help. 0:00 / 4:59

62 daily conference calls Daily conference calls were setup to discuss test findings and build consensus on the severity of each finding.

63 allow for travel time

64 it's just a browser, right?

65 4 ways to test overseas

66 4 ways to test overseas plan it all and hire help

67 4 ways to test overseas keynote (formerly vividence) study

68 44 ways ways to to test test overseas abroad outsource it completely

69 44 ways ways to to test test overseas abroad rogue or ad hoc testing

70 and so, in conclusion... OVERSEAS TESTING PRE- FLIGHT CHECKLIST

71 questions?

72 @joeplanet newtrality.com

73 resources istockphoto Flickr UCD Stencil by Zazelenchuk & Boling Glyphish Icons My Wacom tablet and two hands Created while listening to Master of Puppets by Metallica and Are You Experienced by Jimi Hendrix