Test Automation Event

Transcription

1 1 Test Automation Event

2 Agenda Welcome to Tieturi * Petri Säilynoja / Päivi Hietanen Mark Fewster, Grove Consultants: * Experiences on Test Automation BREAK for coffee & mingle with friends Ru Cindrea, Altom: * Open-source in Mobile Test Automation Toolbox Discussion and wrap-up

3 3 Welcome to Tieturi!

4 Building Capabilities 4 Coaching / Projects Organisation s competencies Basic Intermediat Advanced Personal skills

5 We link ICT to business 5 Projects IT management Processes People Enterprise Architecture IT project Best Practices SW Architecture Software, Applications Lead Develop Use Manage Information Worker Software and tools Infrastructure Services Information Competence for Managers, Developers, Administrators and Information workers

6 Experience Driven Test Automation Mark Fewster Grove Consultants

7 Contents Management issues management support setting appropriate goals large scale issues Technical issues testware architecture life cycle for automated tests standards quality of scripts test effectiveness 7

8 Management support can make or break automation projects need to understand automation capability & limits building good automation takes time and effort and pays the greatest dividends building poor automation can be quick and cheap but costs outweigh benefits longer term need to manage automation projects not every test can or should be automated only automate good tests recognise automation and testing as different skills 8

9 Interesting manager stories the unbelievable case of propaganda Kai Sann: unrealistic goals manager advertised 100% automated testing doomed to fail? the classic case of inappropriate demarcation Bo Roop: automating testing of automation tool technical challenges particularly in maintenance manager: you write tests, others run & maintain but others couldn t maintain, tests abandoned Think of it my way 9

10 Interesting manager stories the unsurprising case of spare time work Ane Clausen: pension system 5 people part time no goals, no time to coordinate, re-invention, not publicised lasted 6 months the mysterious case of death by stakeholders George Wilkinson: NHS Care Records System distributed automation team many challenges overcome just beginning to pay dividends cancelled by stakeholders 10

11 Setting appropriate goals the refreshing case of a clear goal Bryan Bakker: medical devices problems after release, wanted improved reliability a few people, basic execution & check log automated tests found bugs but delayed the release to fix them management reaction? and another Lars Wahlberg: a senior manager set clear goals ( this helped ) provided adequate funds, monitored closely 50,000 new tests in 3 years, reusable for all projects 11

12 CRM system: background and objectives Brian Newman and Ed Allen, USA 5 MLOC, 20 testers (18 auto.), UI testing, TestPartner goal: automation of QA regression tests one of top 3 corporate initiatives for the year mandate to run regression tests: consumed the time automation to free up testers for other high value added testing (exploratory, performance, load) within 1 year: reduce manual regression test time from 10 to 5 weeks, and automated regression from 2 weeks to 24 hours generated excitement and motivation, could track and show progress 12

13 CRM system: challenges gaining developer support (testability issues) sold benefits of faster and more frequent feedback harmony between testers and automators merge roles seeking constant improvement frequent meetings to review what worked and what didn t show-and-tell training team ownership 13

14 CRM system: benefits achieved reduction in regression test time manual regression test time cut from 10 to 5 weeks automated regression cut from 2 weeks to 24 hours no. of customer reported quality issues reduced despite more features and customers others increased DDP from 87% to 97% over 4 releases run automated regression every night on latest build 98 % 96 % 94 % 92 % 90 % 88 % 86 % 84 % 82 % Automation Manual scripted Exploratory Fix verification 9% 24% 58% 9% 14

15 Database testing Henri van de Scheur, Norway tool developed in-house now open source: 9 months, 4 developers in Java (right people) agreed requirements up front earlier time to market consistent repeatable testing reduce cost of testing run tests more often run tests unattended find regression bugs good architecture, started with quick wins flexible config., good reporting, metrics used to improve 15

16 Database testing: results and pitfalls results (after 3 years): 2,400 times more efficient from: 20 people, 40 tests, 1 platforms in 16 days to: 1 person, 200 tests, 6 platforms in 4 day other benefits quick dev. tests, nightly regression, release tests low maintenance, machine use 24x7 better quality pitfalls testing matured at expense of simple things focus on automation at expense of usability 16

17 Contents Management issues manager support setting appropriate goals success on a large scale Technical issues testware architecture life cycle for automated tests standards quality of scripts test effectiveness 17

18 Implementing testware architecture tool will have its own way of organising tests generic solution may not suit will lock you in to that tool a better way (gives independence from tools) organise your tests to suit you, e.g. use pre-processing to copy files to where the tool expects to find them use post-processing to copy result files to where you want them to be it is about making the testers environment ergonomic to work efficiently with minimal mistakes 18

19 Designing testware architecture anecdote Molly Mahai: Test Manager set up good architecture? books not enough picked something to get started after a while, realised limitations too many projects, library cumbersome re-designed architecture, moved things around we didn t know what was needed until we experienced the problems for ourselves redesign quickly small impact learn on the fly versus pilot 19

20 Life cycle for automated tests case study Henri van de Scheur: testing databases, big success life cycle assign different status to tests depending on situation Suspend Development Candidate Active Retired Deleted different types of test suite MATS, Daily, Weekly, Non-Functional 20

21 Standards case studies Mecke, et al: major SAP application, >3 MLOC (auto) Anonymous: large government agency, 12+ automators, 100 s developers/testers coding standards readable and modular to encourage reuse and minimise maintenance effort document standards automated test specifications and reports process standards e.g. automated test development, execution, and maintenance 21

22 Quality of scripts case study and anecdote Mecke, et al: major SAP application, >3 MLOC (auto) Randy Rice, large organisation, big push to automate reviews review by automation expert useful (copied from dev.) daily technical reviews -> peer reviews -> forum static analysis on automation code in-house static analysis tool, routine checks e.g. reuse issues, attribute issues, naming conventions, zombie scripts continually gather new ideas for checks 22

23 Effectiveness of automated tests anecdotes Julian Harty: various experience reports inherited API tests slow to run, days to update poor structure, only thing tested for: run-time exceptions, not does it do what it should deleted half, re-wrote rest based on unit tests networked tests all but one passed it was the only one that actually connected to the server the only one to indicate the truth! others said pass if there was no error message consider whether to improve or discard such tests 23

24 Summary: some key points management issues management support and understanding is essential for long-term success set appropriate objectives and resource, measure the benefits technical issues good testware architecture helps life cycle for automated tests & standards quality of scripts & test effectiveness case studies provide many insights and ideas take advantage of them and improve your own chances of great success 24

25 New book: Experiences of Test Automation 28 case studies + 14 anecdotes collected, commented on and edited by Dot Graham and Mark Fewster contributions from a dozen countries applications from finance to space, IT to embedded both traditional and agile development Model-Based Testing, cloud, exploratory automation, tool-supported manual testing learn from others experiences available everywhere (well, some good places at least!) 25

26 Agenda Welcome to Tieturi * Petri Säilynoja / Päivi Hietanen Mark Fewster, Grove Consultants: * Experiences on Test Automation BREAK for coffee & mingle with friends Ru Cindrea, Altom: * Open-source in Mobile Test Automation Toolbox Discussion and wrap-up

27 27 Free discussion,,,,

28 28 Thank You!