BEHIND CLOSURES. White Paper Authored for: 13th International Conference, QAI

Transcription

1 1 BEHIND CLOSURES White Paper Authored for: 13th International Conference, QAI Author 1: Rajani Parvathaneni Date: 13-Sep Deloitte Consulting India Private Limited Divyasree Technopolis, 124, Yemlur P.O Off Old Airport Road, Bengaluru Author 2: Anup Jaiswal Date: 13-Sep Deloitte Consulting India Private Limited Divyashree Techno polis, 124, Yemlur P.O Off Old Airport Road, Bengaluru

2 2 Abstract Software has become omnipresent and vital in our information-based society, so all software producers. The rapid growth in internet technology has brought with it unlimited opportunities for several industrial innovations. Selling through websites is the fastest growing method of trading worldwide. There are many different types of products and services that are traded on line including books, CDs, cars, holidays, insurance and what not. In response to e-tailing and e-trading, most businesses have now set up their own websites. This in turn has lined up a way for more efficient software products to enable smooth business. Each of these applications must encompass a robust framework so as to conduct the business in a highly efficient way. A thorough testing and analysis is the backbone of a bug free and efficient application. Many projects fail while the rest are completed significantly late, and provide only a subset of the originally planned functionality. And many a times, Enhancements or Product launches will Go- live with the known issues. Why products are preferred to be launched with known limitations? Why not hundred percent testing is carried out and an error free product can be launched? Why exhaustive testing is not feasible? Why testing cycles are always closed when there are known issues or uncovered defects? This white paper in detail covers the various factors and reasons behind these questions. This paper in detail will analyze the driving factors behind closing test cycles and launching the product or feature. Addressed Audience: Test Practitioners and Engineers, Software and Test Managers, QA Managers and Development Managers as well as other professionals interested in building and delivering better software. Objectives: 1. Reasons behind compressed test schedules 2. Alternatives analysis for complete testing 3. Reasons behind launching the products on time with known issues.

3 3 Contents Table of Contents 1. Typical Timelines for Testing: Waterfall model Iterative Model Agile Model Factors Impacting Testing Requirements Phase Design Phase Coding Phase Test Phase factors Why not Exhaustive testing Why not Automation Why Go-Live Case Study Conclusion... Error! Bookmark not defined.

4 4 1. Introduction There is unrest brewing in IT QA departments all over the globe. A frustration borne out of a lack of recognition within their own organization as to the importance of software quality, the potential competitive advantage their company could enjoy if they get software quality right and the major negative impact that can result from poor software quality processes. This whitepaper will look at the issues surrounding software quality in more detail, try to understand corporate attitudes and pinpoint the distinct advantages to getting software on time with reasonable quality for go live versus trying to get 100% quality objective. 2. Typical Timelines for Testing: 2.1 Waterfall model The Waterfall Model was first linear sequential life cycle model to be introduced. It is the major process model which has extensively been used in all typical projects till the advent of iterative models. It is very simple to understand and use. In a waterfall model, each phase must be completed fully before the next phase can begin. At the end of each phase, a review takes place to determine if the project is on the right path and whether or not to continue or discard the project. In waterfall model phases do not overlap. The most evident disadvantage for testing phase in a waterfall model is that any slippage in upward phases will compress the timelines of testing. In current world of software development and testing does waterfall model assures 100% test pass & still meet timelines - No 2.2 Iterative Model Iterative development contrasts with a traditional waterfall method in which each phase of the software development life.the purpose of working iteratively is to allow more flexibility for changes. When requirements and design of a major application

5 5 are done in the traditional method (sometimes referred to as BDUF or Big Design Up Front), there can be unforeseen problems that don t surface until development begins. By working iteratively, the project team goes through a cycle where they evaluate with each iteration, and determine what changes are needed to produce a satisfactory end product.. Problem of compressed timelines for testing couldn t be eliminated even in this model as the untested functionality in initial iterations gets accumulated for the last iterations. In current world of software development and testing does iterative model assures 100% test pass & still meet timelines - No 2.3 Agile Model Agile methodology is an alternative to traditional project management, typically used in software development. It helps teams respond to unpredictability through incremental, iterative work cadences, known as sprints. Agile methodologies are an alternative to waterfall, or traditional sequential development.

6 6 Testing problem couldn t be eliminated even this model. In current world of software development and testing does agile model assures 100% test pass & still meet timelines - No 3 Factors Impacting Testing Times are tough. You know it, I know it, your competition knows it and your boss definitely knows it. With greater competition, more consumer choice and a greater dependency on technology than ever before, every organization faces the challenge of building quality IS systems to support rapidly evolving business requirements. New applications need to get out the door quicker than ever, and you know that the new application upgrade will be hot on its heels. Time is money and departments are constantly looking for ways to cut down their application time to market. On average ~ 40% of total application delivery time taken up with testing, so is it not a wonder that this is an area marked out for special attention? Once a project is initiated on a time line, there can be various Factors that compress the Testing Life Cycle because of which schedules will be missed and there wouldn t be enough time to go and fix the identified limitations on a project. 3.1 Requirements Phase Actual process of gathering requirements will start once the Statement of Work is provided by the client. For signing-up of the contract, SOW will be taken as reference from Client. Once the contract is signed up, detailed requirement analysis will be initiated by the organizations. There could be various factors under this phase which could impact the planned time lines for Requirements analysis phase. Client has not yet decided on logical flow Proposed solution had conflicting flows Dependencies on third party or Interfaces

7 7 Client review and sign off delayed for functional requirements. 3.2 Design Phase The output of the design phase is the system specification. The system specification is like a set of blueprints that the consultants follow when developing your system. And just as it is very difficult, inefficient, and costly to build a house without blueprints, it is very difficult, inefficient, and costly to build a software system without a system specification. Various factors that could impact the timelines of design phase are as follows: Any delay or slippage of Requirements phase Client insisting on a particular technology Client changing decisions on provided requirements Deadlocks in proposed design for a solutions Back and Forth review process of design by the organization and Client Delay in sign-off for design specifications. 3.3 Coding Phase The coding phase of the software life-cycle is concerned with the development of code that will implement the design. This code is written is a formal language called a programming language. Programming languages have evolved over time from sequences of ones and zeros directly interpretable by a computer, through symbolic machine code, assembly languages, and finally to higher-level languages that are more understandable to humans. Various factors that could impact the timelines of design phase are as follows: Ripple effect of Requirements and design phase Design flaws identified during coding Required technical expertise availability Too much complexity of logic Third part tool availability Yet to define Integration points 3.4 Test Phase factors Software testing is a process used to identify the correctness, completeness, and quality of developed computer software. It includes a set of activities conducted with the intent of finding errors in software so that it could be corrected before the product is released to the end users. Various factors that could impact the timelines of design phase are as follows: Ripple effect of requirements, design and coding phases Requirement changes Scope change identified during testing Too many defects in the system Environment availability Test data availability Complex combinations of input data

8 8 All the various factors discussed in above sections lead us to a limited time and resource availability for completing the planned testing. All these factors should be meticulously taken care in order to achieve the planned amount of testing. 4. Why not Exhaustive testing You want better quality software? Right, well it will take more time to get exhaustive testing done. You want the application live next week? OK, but we will either need to double the team of testers or we will only complete 75% of testing and the quality will be compromised. We don't have any more budgets for more resources? Well there is a decision to be made between an on-time application or a high quality application you can t have both. Exhaustive Testing, as the name suggests is very exhaustive. In this type of testing we try to check the output given by the software by entering all the possible inputs, in fact we use all the permutations and combinations of the inputs. Each element of code is verified under this process. This way we are able to find software endurance as well as its ability to handle various types of inputs. Exhaustive testing is not a practical solution because of the following factors: Exhaustive testing does not guarantee error free product. Not Economical. It requires lot of effort in terms of resources and time. Validating every system permutation, input combination and feature would just end up in long schedules which are not worthy against the cost. 5. Why not Automation Can we use it to get testing done faster? Why does testing take so long? Is the question often thrown at the QA department? Well it s a highly manual process even though some departments may use automation tools, it is still a time consuming activity. There is a skill in balancing the required resources, with time pressures, and software quality. Some Organizations feel that 100% automation is a solution for efficient, effective and exhaustive test coverage. But in practicality that is not a viable solution because of the following factors: Automation cannot be attempted unless the application is automation ready. It has been my experience that it is very important to have the application or functionality stable before automation is attempted. For 99% of the applications, it is not possible to achieve 100% automation. Even if it is possible, it does not make any sense from a business case and ROI standpoint When the first build comes out in a sprint/iteration, it will have defects. These defects may be blocking and may not allow for automation script development to proceed. The most critical issue is to find defects and report so developers can fix them quickly and complete the functionality In the case of sprints, the continuous integration builds will not have fully developed features initially. There may be significant changes with

9 9 subsequent builds, including naming conventions for UI elements which will require a lot of rework from the automation team. This can make the automation team frustrated after a few rounds of changes to the same set of scripts. Testers need to validate that all the requirements are implemented as designed and there are no gaps. This will require a manual testing effort for first time. Testing of screen/ui layout is nearly impossible with automation and requires at least one round of testing manually. This will verify that there are no inconsistencies/issues and to certify that everything works fine Automation can only be implemented selectively and the shelf life of a product should be considered before investing. If a simple business application is being developed with little shelf life (say 1-2 years with no potential changes to the application), then it does not make any sense to invest in automation Studies have proven that more than 70% of Test Automation projects fail to deliver their objectives and some of the common reasons are: Objectives are not clearly defined Lack of automation architecture expertise Automate everything Automate very little High cost of automating new functionality Overall to sum up on automation there are various factors that become road block when trying to make 100% automation in testing. Not that it cannot be done but there are limitations when it comes to making automation as primary driver for QA testing. 6. Why Go-Live As per analysis data from a typical organization perspective, projects that go-live on expected date typically have some known minor issues. Is it better to Go-live/Launch the product on a planned date even though there are minor issues? If a customer runs into that specific minor issue wouldn t that cause a bad customer experience? How do we quantify that Bad customer experience to make a call if QA should recommend a GO for go live or hold back. When it comes to go live date it is a very sensitive decision, any changes to it have repercussion to it. There is no direct answer to this but the answer lays in the question itself i.e. Why the application should not go live with that known issues? we need to start quantifying the known issues into factors that can be weighted to make a call. For example consider below factors: 1. Is the issue customer facing or internal user issue? 2. Is the issue going to cause incorrect financial data (prices, statistics) displayed to customer?

10 10 3. Any impact to cash flows (i.e. would it prevent transactions that prevents customer to do business on website)? 4. What is the probability of issue happening on website? 5. Even if the probability of issue is low, if that were to happen then is there any way to alert the user/customer support for corrective action. i.e. what is the ETA of workaround if issue happens on production? 6. Does the issue breaks down any of existing critical process in application with no workaround. 7. Would this issue cause application to crash (no workaround to prevent this from happening if went live)? 8. What is the ETA to fix the issue can it go as patch post go live? Above are just few of suggestive factors to analyze the known issues & the answers would help not just the QA but the project management need to decide, discuss with business to understand the severity of the issue if acceptable for not holding back to go live date. 6.1 Case Study-1 Case is a leading company in manufacturing sector which sells internal units of a product. CASE operates worldwide and about 95 % of CASE sales are reported through their E-commerce portal. After analysis of market and competitors, CASE arrived at a decision that adding one more product to their portal would close the needs of their customers and would generate 20% more revenues to the company. And to retain those customers, they would need to launch the product within 3 months. Based on the above requirement, CASE had come up with the following project plan: Planned:

11 11 Proposed project plan was presented to Vendor Incremental Inc. Incremental Inc is an independent organization and has been working with Case for its e-commerce solutions. Incremental Inc. signs the contract for delivering the project as per the proposed plan and gets into the project initiation. Actual: Case had an issue with design of shipment of new product Case appointed a third party vendor (Biz) to help in storage and shipment of the new product. Biz integration with solution got delayed as the process with Biz started at the end of design phase of solution. Code missed the schedule to get into the SIT because of integration delay. System Integration testing has started late and very less time to complete the planned testing. Additional resources have been added to meet up with the schedule. Defect fixing and regression cycles have to be completed within the available time. Before the gate decision, Incremental Inc just managed to cover the testing of major functionality and has identified some minor issues with delivered functionality. No sufficient bandwidth for development team to fix the issues and for Test team to perform the regression. Gate Decision: To Go-Live on the planned date with known minor issues. Driving factors:

12 12 Since it is a new feature launch, Customer satisfaction index will be at zero. Clients will have the first look/feel of the feature and provide back the feedback which can be utilized in building the enhancements for the feature launched. When further versions of feature launch are announced, customer satisfaction index will keep growing from zero. Client base will be retained. At least half of the revenue targets would be met. Provides advantage over market competitors. Minor or Patch release can be used for the identified minor issues. 7. Conclusion As the experience and data summarizes, complete testing and delivering a defect free product is impossible and not cost worthy. A software product could be launched with known minor issues as it helps in driving the customer satisfaction and revenues. About Authors: Anup Jaiswal System Integration Deloitte Consulting (Direct: Mobile: anjaiswal@deloitte.com Anup Jaiswal is a specialist master with core competency in software testing, test automation, and testability. Familiar with many test tools, he helps teams develop test automation strategies and architectures. He has helped develop automated tests for functional, as well as performance tests for package applications as well as custom development. Anup had been actively involved in competency building in testing COE (Center of Excellence), Deloitte. He has published white papers in both internal, as well as external presentations. Rajani Parvathaneni System Integration Deloitte Consulting Rajani Parvathaneni is working as a Senior Consultant with Deloitte, Bangalore. She has around 7 years of experience in manual and Automation

13 13 testing. She worked on different domains like Telecom, Manufacturing, and Healthcare. She had successfully cleared the foundation level certification in ISTQB, SQL Server 2005 and a certified Mercury 8.2 Product Specialist. She has played different roles like team member, module test lead and release manger. She has experience in co-coordinating Offshore-onsite model and worked with people across India, US, and Germany. References: