We test apps and websites using the power of the crowd. We call that crowdtesting.

Transcription

1 We test apps and websites using the power of the crowd. We call that crowdtesting. You ve crowdsourced your hotel and your taxi, what about your testing? Emily Fielding

2 Hello, TMF! A bit about me: Emily Fielding BSc, BComm, MSc (MiMM) Sales and Account Testbirds Ltd A Testbird for 1.5 years (50% of the company s lifetime) Here to learn A bit about Testbirds: Conducting high quality functional, exploratory and usability testing utilising the crowd Other innovative software testing services Focus on quality and flexibility Customer Satisfaction 9.2 out of 10 2

3 Why wouldn t we use crowdsourced solutions? Impact of crowdsourcing on product quality Ethical implications of engaging crowdworkers instead of employees 3

4 Why would we use crowdsourced solutions? Real people, real perspective Geographically diverse Engaged flexibly Empowering Online management time saving (?) 4

5 Objections to Crowd-sourced Software Testing Quotes from QA Managers/Delivery Managers/Heads of Test I have a testing team of 300 I know our top 20 devices from Google analytics and have bought them We find all critical incidents before launch If there s an error in a feature, I just switch it off so I never have to roll back releases anyway 5

6 Arguments for Crowd-sourced Software Testing Quotes from QA Managers/Delivery Managers/Heads of Test You can t do everything - you have to delegate I try to keep our top 20 devices from Google analytics but a few months later, they re different Our in-house testers are so familiar with the product, we need a fresh set of eyes I didn t think you d find any critical incidents we d missed but you did 6

7 Device diversity Fragmentation within the Android market Source: 7

8 Device diversity Which connected devices are applicable to your product? 8

9 Device diversity Adapting to local language, expectations and network conditions 9

10 My questions to the TMF Over to you Is device diversity a significant challenge for you? Is crowdtesting a solution for that challenge? Where would you, and where wouldn t you use it, and why? If you re experienced with crowdtesting, what has been your experience with Coverage Quality Flexibility Efficiency 10

11 Software Development Life Cycle Where would you use crowdtesting? What? When? How? Which mock-up is better? Is the product concept right? Prototype Feature development Test environment Production Clickable prototypes tested with the target group for UX Is the feature ready to be pushed to test environment? Test-case approach on the feature pass/fail Are we ready for launch? UAT testing across diversity of devices Service desk is getting complaints Exploratory functional testing Beta group testing 11

12 Agile development Testing within sprints Testbirds project manager Feedback from testers? Has acceptance criteria been met? Product Owner 1 Testing requirements / acceptance criteria 2 Testing Example: Duration: Dev.: Test: Go-Live: 14 days Thur 12 PM until Fri 3 PM Mon or later Product backlog (User- Stories) Sprint Backlog 1 2 Definition of testing requirements and test cases based on the user stories in the backlog Development in sprints with different types of testing integrated Functional, exploratory tests Functional, structured tests Deliverable product 13

13 Agile development Testing outside of sprints Prototype evaluation End-to-end tests Evaluation of 2-3 different (click) prototypes with testers Different formats supported: Paper, Pictures, HTML, Detailed report about the favoured prototype and what functionality should be lifted from other prototypes Usability and functional test of your app or web portal and the up and downstream processes with testers Detailed results about order shipping and processing, in-store experience (e.g. interaction with ibeacons) and other offline touchpoints User acceptance tests Competitors analysis Usability tests of your (beta) software product with testers Covering a variety of user stories and scenarios Detailed report about the major usability flaws and recommendations for action for your software, as well as positive feedback Analysis of your (beta) software product with testers compared to 2-4 competitors In and out of industry benchmarks possible Benchmark results from real potential users 14

14 Crowdsourcing dangers and risks The effects of an unregulated industry A lack of regulations Exploitation Ethical Considerations Legality Safety & Privacy Uber & Mechanical Turk Employees or Independent Contractors? Qualifications, Safety, Support, Worker s Rights, etc. 15

15 Crowdsourcing Code of Conduct Creating a safe, lucrative and comfortable working environment The Code of Conduct Respectable tasks only Clarification on legal situations Fair payment Reward Management Clear tasks and reasonable timing 16

16 Crowdsourcing Code of Conduct Creating a safe, lucrative and comfortable working environment Freedom and flexibility Support and feedback Open and transparent communication Best working environment Privacy The Future 17