Stop the Test Automation ROI-based Justification Insanity. Bob Galen President & Principal Consultant RGCG, LLC

Transcription

1 Stop the Test Automation ROI-based Justification Insanity Bob Galen President & Principal Consultant RGCG, LLC

2 Introduction Bob Galen Somewhere north of 30 years experience J Various lifecycles Waterfall variants, RUP, Agile, Chaos Various domains SaaS, Medical, Financial Services, Computer & Storage Systems, ecommerce, and Telecommunications Developer first, then Project Management / Leadership, then Testing Leveraged pieces of Scrum in late 90 s; before agile was Agile Lucent in using Extreme Programming Formally using Scrum since 2000 Currently an independent Agile Coach (CSC Certified Scrum Coach, one of 50 world-wide) at RGCG, LLC and Director of Agile Solutions at Zenergy Technologies From Cary, North Carolina Connect w/ me via LinkedIn and Twitter if you wish Bias Disclaimer: Agile is THE BEST Methodology for Software Development However, NOT a Silver Bullet! Copyright 2012 RGCG, LLC 3

3 Outline Traditional Automation ROI Traditional Business Case Risks & Reality Agile as an Automation Disruptor New Business Case Investments & Selling The Big But Wrap-up Copyright 2012 RGCG, LLC 4

4 It s simple really Manual testing is: Slow Labor intensive Not intellectually stimulating Error prone Iterative Did I say slow? And we re testing software right? So we ought to be able to automate the testing. Copyright 2012 RGCG, LLC 5

5 It s simple really Automated tests are: Fast Low/no errors Free after initial development Run continuously Did I say fast? Did I say free? And we re testing software right? So we ought to be able to automate the testing. Copyright 2012 RGCG, LLC 6

6 Test Automation ROI Sample Points Manual Testing ü Capture size of the team ü Cost per tester usually hourly ü # of Test cases to be run ü Time to run them ü Cyclical nature of the testing cycles (regression, integration, system, etc.) ü Configurations ü Number of tests run per release ü Number of releases per year Do the math to come up with iterative testing run-time investments and costs Copyright 2012 RGCG, LLC 7

7 Test Automation ROI Sample Points Automation Development Then contrast that against the cost for automation: ü Time / cost to establish automation infrastructure ü Time / cost to establish automated tests ü Time / cost for ongoing maintenance and results analysis ü Cost for automation tooling depends on type, ex: Keyword Driven ü Cost per automation developer vs. manual tester ü Automation replacement target for manual test cases Do the simple, discrete math to come up with cost & time to replace manual testing run-time and ROI savings Copyright 2012 RGCG, LLC 8

8 Traditional Business Case Run more tests, faster and with fewer people (lower costs) VALUE being driven by: Running tests! action= reset Did I say with fewer people (testers)? We can hire more developers with the cost savings! And get MORE done We can even reduce the size of the bathrooms J Copyright 2012 RGCG, LLC 9

9 Copyright 2012 RGCG, LLC 10

10 Copyright 2012 RGCG, LLC 11

11 But often there is underestimation of risks So the ROI was always a fuzzy estimate Underestimating the nature of a software project Architecture & design, complexity, risks, unknowns, ambiguity, etc. Underestimating parallel software projects (product + automation) Integration, dependencies, coordination Underestimating ongoing support & maintenance Repairs & updates Designing and adding new tests Impact of new product technologies Underestimating skill levels for automation development Copyright 2012 RGCG, LLC 12

12 It s also very dangerous! Will Robinson It s commoditizing your testing activity. Could you imagine Measuring developer value by LOC and doing ROI? Measuring architectural integrity by complexity analysis and doing ROI? Measuring product owner value by number of backlog elements and doing ROI? Based on some sort of automation introduction? Of course not. It s insane. Then why do we do it with tests and testers? Copyright 2012 RGCG, LLC 13

13 Traditional Test Automation Tools Just a quick stop We re all using them, so they must be good and the approaches must be correct right? However, there are challenges Training Size UI-centric Cost Can t test everything (Back-end database code) One-size Developer / whole team usage Copyright 2012 RGCG, LLC 14

14 Typical Deployment Level of effort Initial design & Architecture Typical Time & Budget Investment Initial Test Case Automation Technology Disruption Ongoing Maintenance Copyright 2012 RGCG, LLC 15

15 Now it may sound Like I m opposed to test automation. Truly, I m not. I think automation rocks! Or like I m opposed to gaining a return I m not Automation is clearly a foundation element for increased quality and going faster releasing more often But, this traditional model or approach of focusing on ROI is sort of insane Copyright 2012 RGCG, LLC 16

16 What agile has exposed to me wrt/automation & Testing Testers vs. a Whole Team view Build teams in ratios developers vs. testers Instead invest in cross-functional teams composed of the skills to design, construct, test, and deliver excellent products Only testers write test automation The entire team writes test automation and, oh by the way, can test as well Only testers test the application The whole team participates in writing / running automation, functional testing, exploratory testing, regression testing, etc. Copyright 2012 RGCG, LLC 17

17 What agile has exposed to me wrt/automation & Testing Traditional vs. Open Source Tools Instead of leveraging singular, UI-centric tools Leverage multiple tools at all-tiers of your application stack Higher cost Lower cost and a sense of Community Developers won t use them; costs Everyone is leveraging the same tool-sets; AND integrating them in Continuous Integration and Continuous Deployment Copyright 2012 RGCG, LLC 18

18 What agile has exposed to me wrt/automation & Testing People are your primary value proposition Instead of looking at testers as a commodity Consider good testers to be as valuable as any other team member Instead of thinking that testing is a by-rote exercise with little intellectual pursuit Testing is a profession. Testers are valuable craftspeople. Testing is an intellectual exercise. Instead of thinking the value is in the test cases The value is in the testers (people) continuously reevaluating the right things to test at the right time for valuable feedback Copyright 2012 RGCG, LLC 19

19 Copyright 2012 RGCG, LLC 20

20 The NEW Test Automation Business Case So, don t Count pennies Count heads Count cycles Work as part of a TEAM Focus in on testing continuously just the right things at just the right time Providing feedback, transparency, and reaction Allowing your teams time to learn and improve Solving your customers problems; delivering value Copyright 2012 RGCG, LLC 21

21 The NEW Test Automation Business Case Run more tests, faster, unattended, continuous feedback, learning & adjustments Yes, saving time. Now what to do with it??? Drive your new VALUE by: Early feedback for adjustments risk avoidance, learning, solving real problems Having more time to continuously refine your test cases (automated and not) Having more time to focus on building Quality into your products Creatively testing the application (Exploratory Testing, Test Idea Generation, Test Design) Copyright 2012 RGCG, LLC 22

22 So, with the additional time INVEST in your testers & quality ü In building In Quality ü In attacking the customers REAL problems ü In better testing ü In more balanced automation ü In team training; in slack time ü In better collaboration ü In doing more than thought possible ü In creative solutions ü In adapting to new technologies ü In having FUN Copyright 2012 RGCG, LLC 23

23 How do we Sell this change? You don t sell it from a testing perspective. Nor from a bean-counting perspective. You allow the results to begin speaking for themselves as we begin focusing on and talking about How one builds truly innovative and creative software teams That understand and solve customer problems With simple solutions that fundamentally work And do so iteratively & quickly, while nimbly adjusting to customer feedback Copyright 2012 RGCG, LLC 24

24 But Bob That doesn't work in the Real World! My boss would never go for it without ROI and hard justification Testing IS a commodity We ve done automation perfectly seeing none of the issues/risks you ve pointed out Copyright 2012 RGCG, LLC 25

25 But Bob Think about what I m saying; taking the time and Invest in people Invest in better quality & testing Invest in better product solutions to customer problems Invest in improved workflow and delivering high customer value WOW how crazy is that? Meet in small groups of 2-3 (pro / con) and chat for 10 minutes about the sides and the possibility for change? Then we ll debrief Copyright 2012 RGCG, LLC 26

26 Wrap-up Hope we challenged your existing assumptions a bit Inspired you to change your view towards Automation ROI and investment What did I miss? Final questions or discussion? Thank you! Copyright 2012 RGCG, LLC 27

27 Contact Info Bob Galen Principal Consultant, RGalen Consulting Group, L.L.C. Director of Agile Solutions, Zenergy Technologies, Blogs Project Times - Business Analyst BA Times - My Podcast on all things agile - Scrum Product Ownership Balancing Value From the Inside Out published by RGCG in Experience-driven agile focused training, coaching & consulting Contact: (919) bob@rgalen.com bob.galen@zenergytechnologies.com Copyright 2012 RGCG, LLC 28