Maximizing the Value of Automation Testing

Transcription

1 Maximizing the Value of Automation Testing November 26, 2013 open forum 1

2 Speaker : Steven Lombardi, M.Eng Senior Manager, Delivery at QA Consultants in June Delivery for all testing and quality assurance projects Future development of the organization and thought leader 17 years in the industry experience in multiple lines of business Major contributor to the success of many organizations both locally and globally in delivering high quality applications on time and on budget Modular and systematic approach combined with his unparalleled industry experience has allowed him to consistently reduce costs, time to market and reduce risk. Prior to joining QA Consultants, Steven was the Automation and Performance Manager for Alliance Data. This position required the designing, budgeting, planning, executing and planned maintenance of an automation framework. Master's Degree in Applied Science in Electrical Engineering from the University of Waterloo with a focus on mobile communications. Honours Bachelor of Applied Science in Electrical Engineering with a minor in Management Science. 2

3 Agenda What to automate How to create a plan How to choose the optimal tool How and why to test early and often How to organize for testing and why it's important How to develop scalable automated tests How to generate quality test data Why you should consider advanced automation tools Question and Answer 3

4 What to automate? Typical Reply: Look and feel out of scope 80/20 rule - Targeted suites (smoke suite, high risk regression suite, etc..) Return on investment is high Ideal test cases: - Repeatable and consistent - Run frequently on multiple builds - Involves copious amounts of data performing the same actions - Run on numerous lab environment combinations - Prone to human error - High risk prioritized 4

5 What to automate? Non Typical Reply: Agile Implement Test Driven Development Use Behavior Driven Development One sprint behind automation development Object Oriented Designed automation Calculate ROI based on future sprints Product demo presented with Automation Mobile Platform independent design Non factory secure device automation Instrumentation of application 5

6 Return on Investment Calculation Assumptions: worst case scenario - web based application testing - test data only used once - one OS and platform used - test work flows are unique - Automation tool is at no cost Legend: CEM = Cost of initial Execution of Manual test cases CMM = Cost of Maintenance of Manual test cases CRM = Cost to Re-execute Manual test cases X = number of re-execution for the project CIA = Cost to Implement Automation CMA = Cost to Maintain Automation CRA = Cost to Re-execute Automation and analyze 6

7 Return on Investment calculation From historical data: CIA = 3 CEM CMA = 4 CMM CRM = 5 CRA therefore CRA = 1/5 CRM ROI for automation = Manual Cost - Automation Cost ( CEM + CMM + CRMx) (CIA + CMA + CRAx) ( CEM + CMM + CRMx) ( 3 CEM + 4 CMM + 1/5 CRMx) -2 CEM 3 CMM + 4/5 CRMx 7

8 Return on Investment calculation example Example situation: CEM = 5 people for 5 days = 25 man days CMM = 5 people for 1 day = 5 man days CRM = 5 people for 3 days = 15 man days x = 2 week builds for 5 months = 10 ROI for automation = Manual Cost - Automation Cost = -2 CEM 3 CMM + 4/5 CRMx = = 55 man days of saving if automation is used 8

9 Create a Plan Architecture is key Service Oriented Architecture Enterprise service bus Database stored procedure Test data setup Database capture and rollback Environment setup Application programming interface Test suite identification Smoke suite, high risk suite, regression suite Application functional areas Development support Changes required for successful automation 9

10 Create a plan Agile Planning session Documentation updates User story details Test data setup Daily scrum meeting involvement Cross team automation development but common framework Definition of done Modular design Mobile Changes required for successful automation Where to obtain and manage the hardware Security state of hardware Wifi communication or Wireless Data Location based services Versions of hardware Version of OS (BB10 not supported) Device contingency 10

11 Choose the Optimal Tool Level of support (freeware vs. purchase) Experience level of work force Lack of functionality due to previous points Code repository Hardware, Operating System and development environment Consideration factors - Object repository overhead - Smart Identification - Built in functions - Browser support and similarities 11

12 Test Early and Often Night build, deploy and execution - Load balancer change, network change, database table change, container configuration change Code check in build, deploy and execute Stable elements (OO design) 12

13 Organize for Testing Internal communication Role and responsibilities defined Automation is a development role Peer coding to ensure success High access levels Application under test expert Test case, test data, steps, validation points clear 13

14 Develop Scalable Automated Tests Independent fundament components Designed for change Lower maintenance Accelerate script development Unique object identification through company Uniqueness is retained (build to build) 14

15 Develop Scalable Automated Tests Typical Reply: Use common actions - A grouping of common sequence of events - Parameters are either local or global Common advancement - Pass parameters in Common example - Login action - Passing in the username and password 15

16 Develop Scalable Automated Tests Non - Typical Reply: (applies to both agile and mobile automation) Turn actions into Functions - Allows for over loading - Allows for return and report - Allows for validation point and easy of debugging Break common actions into smaller elements - Using example: from QTP 16

17 Develop Scalable Automated Tests SystemUtil.CloseProcessByWndTitle "Flight Reservation" SystemUtil.Run "C:\Program Files (x86)\hp\quicktest Professional\samples\flight\app\flight4a.exe",".","C:\Program Files (x86)\hp\quicktest Professional\samples\flight\app","open" Dialog("Login").WinEdit("Agent Name: ").Set "John" Dialog("Login").WinEdit("Password:").SetSecure "528a ae81ef0db32c05def8445db11646b3642" Dialog("Login").WinButton("OK").Click var_exist = Window("Fight Reservation").Exist(60) If var_exist <> True Then ' ExitRun End If ExitTest("Could not load the Fight reservation windows even after a wait of 60 seconds")

18 Develop Scalable Automated Tests SystemUtil.CloseProcessByWndTitle "Flight Reservation" SystemUtil.Run "C:\Program Files (x86)\hp\quicktest Professional\samples\flight\app\flight4a.exe",".","C:\Program Files (x86)\hp\quicktest Professional\samples\flight\app","open" Dialog("Login").WinEdit("Agent Name: ").Set "John" Dialog("Login").WinEdit("Password:").SetSecure "528a ae81ef0db32c05def8445db11646b3642" Dialog("Login").WinButton("OK").Click var_exist = Window("Fight Reservation").Exist(60) If var_exist <> True Then ' ExitRun End If Create a function to close all unwanted applications Pass in a white list and black list Include a validation ExitTest("Could not load the Fight reservation windows even after a wait of 60 seconds")

19 Develop Scalable Automated Tests SystemUtil.CloseProcessByWndTitle "Flight Reservation" SystemUtil.Run "C:\Program Files (x86)\hp\quicktest Professional\samples\flight\app\flight4a.exe",".","C:\Program Files (x86)\hp\quicktest Professional\samples\flight\app","open" Dialog("Login").WinEdit("Agent Name: ").Set "John" Dialog("Login").WinEdit("Password:").SetSecure "528a ae81ef0db32c05def8445db11646b3642" Dialog("Login").WinButton("OK").Click var_exist = Window("Fight Reservation").Exist(60) If var_exist <> True Then ' ExitRun End If Create a function to open required components Pass in location of application Pass in browser under test and set global variable for remainder of test (example: local language) Include a validation ExitTest("Could not load the Fight reservation windows even after a wait of 60 seconds")

20 Develop Scalable Automated Tests SystemUtil.CloseProcessByWndTitle "Flight Reservation" SystemUtil.Run "C:\Program Files (x86)\hp\quicktest Professional\samples\flight\app\flight4a.exe",".","C:\Program Files (x86)\hp\quicktest Professional\samples\flight\app","open" Dialog("Login").WinEdit("Agent Name: ").Set "John" Dialog("Login").WinEdit("Password:").SetSecure "528a ae81ef0db32c05def8445db11646b3642" Dialog("Login").WinButton("OK").Click var_exist = Window("Fight Reservation").Exist(60) If var_exist <> True Then ' ExitRun End If Create a function to locate elements returning object Create a function to fill in text fields (WinEdit) Include a validation ExitTest("Could not load the Fight reservation windows even after a wait of 60 seconds")

21 Generate Quality Test Data Repeatable independent data Potential automation of data setup Database snapshot and restore Storage location of test data; version controlled Consider Advanced Automation Tools Business processes automation Behavior-Drive Development Different frameworks for success 21

22 BDD Template Based on User Story As a [Role] I want [Feature] So that I receive [Value] Given [Context] When [Even Occurs] Then [Outcome] 22

23 Conclusion Automation can improve testing outcomes Design is required for optimal efficiency Tool, People and structural model are key 23

24 Presenter's Contact Information Steven Lombardi M.Eng Senior QA Consultants Tel: slombardi@qaconsultants.ca Available also on Linkedin : ca.linkedin.com/pub/steven-lombardi-m-eng/0/38b/2b/ 24

25 Question and Answer Contact Information 25