Defect Intelligence A Must-have for Building World-class Quality

Transcription

1 Defect Intelligence A Must-have for Building World-class Quality By Karthik Padmanabhan Karthik.padmanabhan@cognizant.com Karthikeyan Shanmuga Shanmuga.karthikeyan@cognizant.com Cognizant Technology Solutions, Techno Campus 5/535, Old Mahabalipuram Road, Okkiyam, Thoraipakkam, Chennai , India Defect Intelligence Cognizant Technology Solutions Page 1

2 Defect Intelligence: A Must-have for Building World-class Quality ABSTRACT: Traditional testing units in an IT services organization are primarily involved in testing the application/product as per the stated requirements and report out the defects identified in their various test cycles. The primary function is merely effectiveness and efficiency in defect identification. However, this may not cater to the dynamic needs of the business transformation in the current world scenario. The customers today are more demanding and are looking at solutions to build in quality, rather than just identifying and removing defects prior to Go-Live. The key things which the customer looks up to include: Identify risks on the product quality & application stability with the help of data generated during the test lifecycle To provide value to customers by providing necessary intelligence on the quality of the application in the software development lifecycle Partner with the customer in aligning to their business goals by reducing the total cost of development with higher predictions on application stability with specific interventions Testing organizations, by virtue of having access to enormous amount of defect data, are ideally poised to provide such diagnosis and make recommendations to clients on how they can engineer quality into their products much earlier in the lifecycle. Having performed such defect analysis on several of our clients applications and products, this paper presents inferences, trends and recommendations to build high quality products. This analysis provides deep insights into: Source and origin of such defects Top contributing factors to defect leakage across phases Application defect proneness and application stability Building investment business cases for shift-left strategies and implementation of Optimization techniques leveraging Lean, Cost of Quality & Six Sigma for strengthening the SDLC Defect Intelligence Cognizant Technology Solutions Page 2

3 Need for Defect Intelligence In today s rapid changing Dynamics of the IT industry, the cost of identifying and fixing defects or bugs in the single largest cost element in the entire software development and maintenance life cycle. Research proves that 50% of the overall cost of software development is attributed to identification and fixing of defects. The cost of identification and fixing the defects increases exponentially in the final stages particularly in the downstream activities such as user acceptance testing, bugs in production and also amounts to greater risk to the project in terms of schedule. A little extra amount of effort spent on quality assurance will see good amount of cost savings in terms of detecting and eliminating the defects. Some of the key Pain areas of the customers include: There is significant difference between budgeted cost and actual cost The application quality is not predictable There are schedule slippages for most of the releases There is too much time and effort spent on rework Production defect leakage continues to occur even after rounds of testing Customers today are looking at their vendor partners to bring in key value differentiators in terms of lower total cost of ownership, improving the time to market of the application with faster turnaround times and with built-in quality assurance. Today testing organizations have good amount of quality data on the defects of various releases and builds available in their individual defect management tools. It is time that the testing teams not just measure the effectiveness and efficiency of the testing process but also look beyond in identifying patterns on the defects in the overall software development life cycle and attempt to provide specific interventions which will help in improving the defects not getting leaked to subsequent stages of the SDLC. The overall intelligence provided enables today s testing organization transition from pure play testing services to mature Quality Engineering and assurance services. To gain a deeper insight of the effectiveness of the software process, it is essential to examine the details of the defects detected in past releases understand the root causes for the defects and help in predicting how the defects getting leaked could be arrested there by reducing the overall cost of defects getting leaked into subsequent stages. Quality Engineering Diagnostic (QED) is a quick quantitative approach as part of the overall Quality Intelligence for assessing the effectiveness of SDLC process to identify key focus areas and thus unearth defects as early as possible Defect Intelligence Cognizant Technology Solutions Page 3

4 Quality Engineering Diagnostic is a quick check quantitative tool founded on metrics that can help quickly establish the quality of the Software Development Lifecycle (SDLC) and help in aligning for specific Interventions for strengthening the SDLC Enable Shift-Left strategy Establish the relative quality of key functions across the SDLC Establish the potential impact on Cost, Quality & Time leveraging process optimization techniques Identify the possible areas where intervention of the management team is required Identify possible areas where a deeper analysis is required to identify the root causes Testing organizations has access to information on the defects captured in the various stages of SDLC. Typically most of the information resides in the defect management systems maintained for the project. QED helps in identifying the Key metrics which can be generated from the available data. As a part of the QED exercise we did a study to understand the different levels of information which are being maintained in the defect management systems across various client projects. The Key data available on the defects from most of these systems include: Defect ID Test cycle information Severity & Priority of the Defect Stage of identification of the defect Stage attributed to the cause of the defect Defect cause Defect status, including defects which get re-opened due to bad fixes Defect Time stamps (Open date, Fixed date and re-opened date ) Detailed description of the defect Effort information taken to execute and test cases Effort information on the time taken to fix the defects With these available information, key Metrics have been identified which will help in establishing the quality of all functions across SDLC. Metric name Defect injection distribution Defect Trend by classification Defect detection distribution Defect Leakage Defect Density Defect Ageing Defects By Application Defects By build Requirements to resource ratio Application stability Rework development effort% Quality of Fixes Bug Rejection % Operational Definition Count of Defects injected in a particular phase of SDLC to total defects Classification of defects by Type and cause Distribution of defects detected during the various stages % of defects leaked from one stage to another stage based on overall defects Number of defects per total effort spent (weighted) Average time taken to close a defect Total Defects to each application serviced Total defects to each build released Number of requirements executed by the team # of postproduction defects to application size Rework effort spent for fixing defects identified during testing phase Number of reopened defects as % of total fixed defects Ratio of defects rejected to total defects Defect Intelligence Cognizant Technology Solutions Page 4

5 A few of the critical metrics which make a difference in studying the quality of SDLC include defect Leakage, defect detection and injection trends and turnaround time to close a defect in addition to the traditional project metrics. These metrics helps us to narrow down to the stage level pain areas which need to be looked upon for specific interventions for improvement. Defect Leakage: This metric helps in identifying the percentage of defects getting leaked at every stage of the SDLC lifecycle. This helps us to understand and review the current review mechanisms Defect Injection and Detection trend: Both these metrics helps us to understand the trend of defects being injected and the current project team s capability in detecting the defects at the stage of occurrence. This is a key metric in determining the review efficiency of the team. With this information we can provide key value to the customer in strengthening the software development activities. Defect Turnaround time: This metric helps us to understand the average time taken to close the defect. Longer the cycle helps us to narrow down on the maturity of the development process for fixing the defects. This metric is a key input for gauging the quality of the development team in churning out the fixes for timely delivery. Approach for Administering QED QED is very effective for execution in long running engagements where there are periodic releases being carried to the application. Historical Defect details for at least a year encompassing of at least 4 releases (Quarterly releases) are taken from the defect management tool. Using the defect details trends and patterns are derived using the key metrics as part of the QED and a summary report is prepared for identifying specific interventions leveraging Shift left, reducing cost of quality Lean, Six-sigma and other process optimization techniques for overall quality improvement. The High level steps for administering QED include: Data Collection and Consolidation - through the defect management system Data analysis and interpretation on the key metrics from QED Summary report with specific interventions for improving the overall Quality QED acts as a base for the project managers to identify and propose meaningful and systematic data driven technique for holistic problem solving by bringing in adequate intelligence from the available defects data. QED Outcome: We had conducted extensive study of QED by piloting it across our banking, insurance and retail customers. QED had generated lot of interest amongst them in helping them with specific interventions for strengthening their quality of the application. Defect Intelligence Cognizant Technology Solutions Page 5

6 Some of the key Findings from that exercise include Ageing of Open Defects were considerably high across projects o There was no significant difference in the ageing of defects by the severity excepting for showstopper defects which were being addressed immediately o The average time to close High and medium category defects was similar to those of Low categories and they were very less prioritization being made o In many cases due to schedule crunches, new builds was being churned even before the fixes were complete resulting in higher Defect Distribution of defects by various Defect Categories indicates systemic gaps in SDLC process resulting in Defects getting leaked to later stages o Coding and environment related defects are significantly higher across projects being analyzed resulting in poor quality of code and in effective build management being followed o Other observation being frequent changes to requirements not being managed adequately resulting in higher defects being unearthed Injections of Defects in the early stages of the lifecycle are being identified only in QA leading higher cost and time taken to fix the defect. o There were no evidences of proper defect management process at the development end o Multiple releases / tracks being managed by the same development teams resulting in process gaps in release The overall Defect containment at the development seems lacking across releases leading to Higher costs to the clients o Lack of robust review mechanisms including non-implementation of automated code reviews and unit testing were observed o Less evidence of Defect prevention mechanisms including basic RCA being carried out after every release leading to repetitive defect across releases QED Challenges: One of the challenges which we faced during our initial pilot was availability of adequate information in the defect management systems. This could be a deterrent for QED. It is necessary that we select those projects where testing activity is matured and there is a process to capture the defect information preferably using a defect management tool. QED Case Study Leading Banking customer The customer was a leading bank in the US and we were looking at the application landscape where testing services were carried out by Cognizant and development of these applications was carried out by other vendors. We had identified certain Key business lines of the customer under the investment banking domain for our analysis, taking into account - Time to market being a key focus area of the QED. Defect Intelligence Cognizant Technology Solutions Page 6

7 Corporate Corporate Corporate Some of the charts based on our analysis are depicted below: Defect Injection and Detection capabilities % Defect Injection 0.3% % Defect Detection Technologies Treasury Services Grand Total 3.3% Java and Tandem 7.2% Java and Oracle 2.9%.NET/Java/Tandem/Mainframe 2.4% 6.9% 19.6% 17.4% 28.4% 10.2% 9.9% 19.7% 7.3% 62.8% 54.0% 43.3% 50.9% 2.0% 7.2% 3.9% 0.5% 5.9% 6.1% 0.4% 5.9% 4.2% 0.7% 5.0% 9.6% 2.5% Technologies Treasury Services Grand Total0.0% 0.1% 0.3% Java and Tandem0.1% Java and Oracle 0.5%.NET/Java/Tandem/Mainframe 0.1% 0.3% 0.0% 24.3% 38.8% 67.5% 72.1% 42.6% 21.6% 0.0% 24.7% 32.3% 11.9% 37.6% 8.0% 0.0% 15.5% 1.6% 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0% 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0% 0- Planning 1 Requirements 2 Design 3 Coding Environment Issue Inherent Defect Not a Defect Test Data Issue 1-Requirements 2-Design 3-Development 4-SIT 5-QA 6-UAT 7-Production % Developer's rework by Defect Detection Technologies Treasury Services Grand Total 2.0% 6.3% Java and Tandem Java and Oracle 2.6%.NET/Java/Tandem/Mainframe 48.5% 70.4% 50.8% 66.4% 8.2% 9.8% 0.0% 6.2% 33.4% 29.6% 38.4% 27.4% 0.0% 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0% 1-Requirements 2-Design 3-Development 4-SIT 5-QA 6-UAT 7-Production The above charts clearly provide us indications of the defects are not getting identified significantly till QA and as a result there is also a spillover of defects to UAT and production as well. As a result the effort spent by developer for fixing the defects detected at a later stage is significantly very high. (E.g.: 8% of the defects detected in production consume nearly 33.4% of the developers overall rework effort). This indicates the poor quality of defect containment activities being carried out at the development end, which could be looked upon as an area of focused assessment. In addition to this across the LOB we noticed that the turnaround time for a defect getting closed was considerably very high. The average ageing for was ~ 20 days for 50% of the applications serviced. There were also trends which had indicated that there was lack of prioritization of defects based on severity and priority and as a result critical defects were taken more time to fix when compared to less critical defects. Defect Intelligence Cognizant Technology Solutions Page 7

8 QED helped us to unearth lapses in the early stages of the development lifecycle which was resulting in delayed deployment resulting in higher cost to the customer. Specific Interventions leveraging Shift Left and Lean were proposed to the customer for improving the defect containment at the early stages which could result in reducing spill overs in the overall IT budget for the customer. Conclusion: QED thus helps in redefining the testing organizations strength in the intelligence they could bring to the table with the availability of vast testing data and help in redefining themselves from mere Testing team to Quality Engineering and Assurance teams from the eyes of the customer. This change will help the customers in the long run in reducing the Total cost of ownership of the applications Defect Intelligence Cognizant Technology Solutions Page 8