NUANCES OF RELEASE REGRESSION QA METRICS A QUANTITATIVE OUTLOOK VENKATESH JAYARAMAN QA MANAGER RAM KUMAR CHANDRASEKARAN SR DEV MANAGER HIGHRADIUS

Transcription

1 NUANCES OF REGRESSION QA METRICS A QUANTITATIVE OUTLOOK VENKATESH JAYARAMAN QA MANAGER RAM KUMAR CHANDRASEKARAN SR DEV MANAGER HIGHRADIUS TECHNOLOGIES 1

2 Abstract Release Regression testing has taken precedence amongst various testing types in the current Agile QA realm. Testing levels have seen some critical changes, especially with the advent of Agile project management and their methodologies. Continuous Integration, Release Level Regression (RLR), Automated Unit Testing, Test Driven Development (TDD) have taken over the traditional Functional, System, System Integration types of testing. Although the focus was and is to perform Black Box Testing, this new direction in test strategies brought about a change in the way of measuring testing metrics. Software testing metrics often is overlooked with the Dev-QA collaborative work style of agile projects. Burndown chart, Velocity, Backlog items etc., do not emphasize the testers efforts. It, hence, becomes implicit to define, capture, measure and decide QA metrics at least at the pre-production gate or otherwise called the Release Regression gate with a focus of basic and combined testing metrics. This paper will throw light on some of the key derived QA metrics at Release Regression level, that have stood as decision-makers-compass and enabled Testing leadership/management, Release and Change management board, Executive board to take a judicious go/no-go call(s) during Agile deliveries. 2

3 CONTENTS TESTING LEVELS TRADITIONAL Vs AGILE WORLD QA METRICS - A DEBRIEF STABILITY QUOTIENT - IMPORTANCE RSQ QUANTITATIVE DERIVATION AND CASE LET CONCLUSION 3

4 Test Levels Sprint Planning Release and Sprint Backlog TESTING LEVELS TRADITIONAL Vs AGILE Development Unit Testing Types of Environments System Integration Test Functional & Integration Non Functional SPRINT (Crossfire or CI) Unit/Acceptance Tests Continuous Integration Testing Functional & Non Functional Testing Continuous Regression Testing Stage/UAT Acceptance Testing Stage/UAT Sanity Testing Release Regression Parallel Testing Traditional World Agile World 4

5 QA METRICS A DEBRIEF QA Metrics Traditional World Long Term, 33 Medium Term, 22 + Short Term 15 Measures, 55+ Metrics 54 Process, 16+ Product based 57 Tactical Metrics QA Metrics Agile World RTF Metrics 16+ Code and Complexity MEtrics 15+ Volumetric s, CI and Defects based MEtrics 26+ Strategic 33 Operational 5

6 Release Stability Quotient Importance Execution Success Index Release Stability Quotient AGGREGATE OF KEY DERIVED METRICS WEIGHTED AVERAGE Execution Others STABILITY QUOTIENT Regression Automation Test Case Stability Index CUSTOMIZABLE TO PROJECT/PRODUCT NEEDS VALUE SCORE PAIR BASED CALCULATION DEPICTS THE CONFIDENCE OF THE AIDS IN BUILDING A TREND ACROSS S RELATIVE CHECK POINT OF VARIOUS S SERVES AS PRE-PROD QUALITY CHECK Business Risk Index 6

7 Execution Execution Success Index Regression Automation EXECUTION COVERAGE SERVES AS EXIT CRITERIA FOR REGRESSION TESTING HELPS IN TEST EXECUTION PLANNING FOR SUBSEQUENT S TACTICAL METRIC Others STABILITY QUOTIENT Test Case Stability Index LAGGING METRIC INDICATING TESTING QUALITY OF THE PROJECT/PRODUCT WEIGHTAGE CAN BE INCREASED FOR TESTER/QA FOCUSSED Business Risk Index 7

8 Execution Execution Success Index STABILITY QUOTIENT Regression Automation EXECUTION SUCCESS INDEX METRIC TO DEPICT QUALITY OF CODE HELPS IN COMPARITIVE ANALYSIS OF PRODUCTS AND THEIR CODE ACROSS S METRIC TO DEPICT THE EFFECTIVENESS OF FUNCTIONAL TESTING TEAM WEIGHTAGE CAN BE INCREASED FOR QUALITY CODE/DEVELOPER FOCUSSED Others Test Case Stability Index Business Risk Index 8

9 Execution Execution Success Index STABILITY QUOTIENT Regression Automation REGRESSION AUTOMATION COVERAGE HELPS IN FASTENING THE TEST EXECUTION CYCLE EXPEDITES AN EARLY TESTING-ROI INCREASE OVERALL TEST CASE COVERAGE TACTICAL AND LEADING METRIC TO DETERMINE QUALITY WEIGHTAGE CAN BE INCREASED A ROI FOCUSSED Others Test Case Stability Index Business Risk Index 9

10 Execution Success Index TEST CASE STABILITY INDEX STRATEGIC METRIC Execution Others STABILITY QUOTIENT Regression Automation Test Case Stability Index PROCESS ORIENTED METRIC CONTROLS TEST CASE DESIGN EFFECTIVENESS OF FUNCTIONAL TESTING TEAM EMPHASIS ON PEER REVIEWS WEIGHTAGE CAN BE INCREASED FOR TESTER/QA FOCUSSED Business Risk Index 10

11 Execution Success Index BUSINESS RISK COVERAGE INDEX STRATEGIC METRIC Execution Others STABILITY QUOTIENT Regression Automation Test Case Stability Index LEADING METRIC TO DEPICT PRODUCT QUALITY PRODUCT EMPHASIS HELPS IN PRIORITIZING IMPACT PRONE BUSINESS AREAS EFFECTIVENESS OF TEST PRIORITIZATION TECHNIQUES LIKE RBT ALIGN BUSINESS PRIORITIES HELPS IN DECISION MAKING AT LEVEL WEIGHTAGE CAN BE INCREASED FOR BUSINESS/SME/PRODUCT OWNER FOCUSSED Business Risk Index 11

12 Execution Success Index OTHERS # OF DEFECTS OPEN Execution Regression Automation ENVIRONMENT ISSUES NETWORK DOWNTIME STABILITY QUOTIENT PLANNING ISSUES/RISKS Others Test Case Stability Index Business Risk Index 12

13 QUANTITATIVE CASELET 13

14 REFERENCES & APPENDIX s/ibm.pdf pdfs.s3.amazonaws.com/de18/137df9ddb5cfbfc3922e516524b7750f413c.pdf

15 CONCLUSION A judicious and dedicated effort to capture test measures and metrics has always been appreciated. It is high time for the Agile QA community to comprehend the fact that metrics like Velocity, Burndown chart, User Story point accomplished etc., are tactical and should give way to the world of nested-derived metrics like RSQ to drive Agile releases. A conscious effort, hence, has to be made during Release Planning to capture Release Stability Quotient (RSQ) and adhered to as the pre-prod checkpoint. A trend over releases accompanied with a customized information gathering can aid management take key release level decisions. 15

16 AUTHOR BIOGRAPHY Venkatesh Jayaraman serves as a Quality Assurance Manager with HighRadius Technologies. With more than 11 years of experience in Software Testing Industry, Venkatesh primary areas of interest include Test Process Improvement, Test Management, Implementing Agile QA practices/processes, Test Maturity assessments, Test Metrics and Measurements. He holds a Bachelor s degree in Engineering from Anna University. Venkatesh has been instrumental in delivering some key testing engagement on Agile methodologies with emphasis on Release Regression (Agile-Scrum), Stage and End to End testing types. He holds certifications, from proficient IT bodies, including PMI- ACP, PMI-PMP, ISTQB CTFL, CTAM, TMMi, AICPCU-AINS and various relevant industry vertical certifications. Ram Kumar Chandrasekaran is a Senior Development Manager for one of the largest selling product teams of HighRadius Technologies. Ram has a total of Nine years software industry experience, four of which have been in the roles of managing teams and delivering software. Ram is a gold medallist from IIM Indore in the flagship PGP Programme ( ), and has been a practicing manager since. He has immense knowledge and hands on experience in Agile software delivery methodologies. During his tenure with Wipro Technologies ( ), he gained rich exposure in the design and development of software applications 16

17 Thank You!!! 17