Session Topic: Achieve Quality Excellency by Engineering Test Strategy for Custom Development Projects

Transcription

1 Session Topic: Achieve Quality Excellency by Engineering Test Strategy for Custom Development Projects Conference Name: 13th Annual International Software Testing Conference (STC 2013) Author Name: Mohan Kumar B S Mail Id: mohan.kumar.b.s@sap.com Company: SAP Labs India Private Ltd, #138, EPIP, Whitefield, Bangalore

2 Abstract In today s competitive world, it is highly challenging to achieve customer satisfaction. Main corporate mantra is to achieve customer satisfaction is by reducing TCO and achieving high quality. Way to achieve high quality is through quality testing. Typically custom development projects follow different SDLC processes based on the certain criteria s. It is very essential to engineer or tailor effective test strategy and test planning to achieve the product quality, in turn achieve the highest customer satisfaction. This session mainly focuses on best practices of test strategy and planning such as: Flavors of custom development projects and quality requirements Developing test strategy and planning based on the SDLC process Early testing and review techniques Adopting value stream mapping Effort estimation, resource planning and risk management in testing What could be the best automation approach and when to plan Introduction to effective intuition based testing like exploratory testing and how to plan/design Defect consignment techniques Effective reporting and escalation process Test case designing with appropriate techniques How to achieve quality readiness before customer acceptance How to adopt scrum at testing Case study on how SAP Custom Development has adopted test strategy and planning processes for customer projects.

3 Many of enterprises run globally. So called, enterprise application is a business application, obviously a big business application which are very much needed for enterprises to run their business in scalable manner. In today s corporate environment, enterprise applications are complex, scalable, distributed, component-based, and missioncritical. They are data-centric, user-friendly, and must meet stringent requirements for security, administration, and maintenance. In short, they are highly complex systems. Designing and developing such enterprise applications means satisfying hundreds or thousands of separate requirements. What s more, every development decision you make to satisfy each requirement affects many other requirements, often in ways that are difficult to understand or predict and the failure to meet any of these requirements can mean the failure of the entire project! Achieving highest customer satisfaction is important aspect of every application development. In today s competitive world, it is highly challenging to achieve customer satisfaction. Main corporate mantra is to achieve customer satisfaction is by reducing TCO and achieving high quality. Way to achieve high quality is through quality testing. Typically custom development projects follow different SDLC processes based on the certain criteria s. It is very essential to engineer or tailor effective test strategy and test planning to achieve the product quality, in turn achieve the highest customer satisfaction. Many enterprise applications are built on standard requirements or as per the industry specific standards. Implementing such enterprise applications means there can be N number of new requirements come up to satisfy the specific enterprise business needs. Volume of custom development projects can vary from small to XXL.

4 Typical custom solution life cycle as follows: Some of the Software Development Life Cycles follows: Iterative Waterfall: Lean Development:

5 Agile Development: Engineering effective test strategy is very much essential. A test strategy is an outline that describes the testing approach of the Software development life cycle. It is created to inform project managers, testers, and developers about some key issues of the testing process. This includes the testing objective, methods of testing new functions, total time and resources required for the project, and the testing environment. Test strategies describe how the product risks of the stakeholders are mitigated at the test-level, which types of test are to be performed, and which entry and exit criteria apply. They are created based on development design documents. System design documents are primarily used and occasionally, conceptual design documents may be referred to. Design documents describe the functionality of the software to be enabled in the upcoming release. For every stage of development design, a corresponding test strategy should be created to test the new feature sets. Defining Test Objective, Test Scope and Project Methodology: Define the purpose for designing and executing a test. Test Scope is based on the specification. Test Scope includes New Scope, Out of Scope and Existing Functionalities from the standard software. It will be appropriate to explain the project methodology followed. This will in turn decide the test planning. Define Test Levels/Types: The test strategy describes the test levels to be performed. Test 1: Static Test Test 2: UT (Unit Testing) Test 2: MIT (module Integration Testing) Why Static Test?:Static testing is performed to prevent bugs early in development cycle. Implementation approach: Test Owner: Tools/Technique: Why UT?:Unit testing is performed to determine that individual program modules perform as per the design specifications. Goal: The purpose of the Unit Test is to check and verify that development objects are behaving as per the design and Requirement Specifications. Implementation approach: Test Owner: Test System: Tools/Technique: Link to Unit Test Cases: Link to Unit test results: Why MIT?:This is performed to determine that how well the program modules get along. Goal: is to test the cross-functional integration points as well as end-to-end business processes. Scope: The scope will cover all development as per the SOW. Implementation approach:

6 Test 3: ST (System Testing) Test 4: AT (Customer Acceptance Test) Test Owner: Test System setup /Test data: Test Resources: Tools/Technique: Link to MIT Test Cases: Message Handling: Why ST?:This is performed to determine that how well the application works in integrated system. Goal: Testing an integrated system to verify that it meets specified requirements. Scope: The scope will cover all development as per the SOW. Implementation approach: Test Owner: Test System setup /Test data: Test Resources: Tools/Technique: Link to ST Test Cases: Message Handling: Goal: To gain acceptance of all functionality from the customer. Customer Acceptance Test shall verify that the system meets customer requirements as specified. Scope: < as discussed with customer and based on the specification document> Test Owner: <point of contact provided by customer> Message Handling: Test Design Techniques: The objective of using Test design techniques is as follows: Understand and apply test techniques during design or test execution Learn to choose appropriate testing techniques in specific circumstances Using these techniques internally will lead to better tested software and have a positive impact on efficiency and effectiveness of testing resulting in higher quality of our projects / products. The choice of which test techniques to use depends on a number of factors: Customer requirements level/type of risk knowledge of the testers time and budget previous example of type of defects found development life cycle, use case models Test Key Idea Method Applicability Design Techique All-Pairs Test combinatorics involving a large set of parameters Systematic "paircoverage" of large parameter space - find 70-90% of defects while reducing number of possible test combinations Parameters semicoupled Large number of parameters and values

7 Boundary Value Analysis Decision Table Equivalence Class Partioning Optimized test cases / test coverage: Test boundary value conditions in addition Specify exhaustive test by table Optimized test cases / test coverage: Single value represents the whole set Find boundary values which generally have a high likelihood of defect occurrence Full spec of input- >output mapping Exhaustive test of all combinations Find values that are "treated the same" and only test these without loss of fidelity in test Any range/equivalence class All parameters are independent Small number of parameters and values Parameters with large value/set range Intuitive Test Approach: Plan appropriate intuitive test approaches based on the situation. Example - Exploratory Testing. These kind of test approached can identify the hidden issues, which could not be identified by normal test approached and standard test cases. Roles & Responsibilities: The roles and responsibilities of test leader, test manager, individual testers, project manager, quality manager are to be clearly defined. This may not have names associated: but the role has to be very clearly defined. Testing strategies should be reviewed by the developers. They should also be reviewed by test leads for all levels of testing to make sure the coverage is complete yet not overlapping. Both the testing manager and the development managers should approve the test strategy before testing can begin Role Project Lead SAP Test coordinator SAP Architect / Technical Lead Tester Developer Responsibility with regards to test activities Primary contact between the Customer and the Project development team Primary contact for all change control requests (if any) Primary contact for all testing activities Responsible for coordinating the test activities, resources and reports at SAP Responsible for implementing the Defect Management Process together with Customer Responsible for the development activities within the project Primary contact for test cases/scenarios Authorized to set and change the defect fix priorities with the customer s agreement during defect review meetings Responsible for proving each transport to ensure that it is stable enough for testing activities to proceed Accountable for message dispatching Executes the test cases and provide test feedback in accordance with the agreed tools Submits detailed information regarding issues and defects Verifies defects that have been fixed Responsible for reviewing defects and for giving accordant status feedback Developers are required to resolve defects in accordance with the project coding

8 System administrator Project Lead Customer Test coordinator Customer Subject Matter Experts Tester System administrator standards Responsible for availability of the development and test systems Primary contact between the test group and SAP Primary contact for all change control requests (if any) Primary contact for all testing activities Responsible for coordinating the test activities, resources and reports at Customer Create test cases that validate listed functions on the Specification, based on Customers business process scenarios. Data customization tasks for the development / test systems. Creates and input configuration, master data, transactional data and pre-staged test data. Executes the test cases and provide test feedback in accordance with the agreed tools. Submits detailed information regarding issues and defects Verifies defects that have been fixed Responsible for availability of the development and test systems Testing Tools: There are two methods used in executing test cases: manual and automated. Depending on the nature of the testing, it is usually the case that a combination of manual and automated testing is the best testing method.

9 Automation: Think of ROI Risks & Mitigation: Any risks that will affect the testing process must be listed along with the mitigation. By documenting a risk, its occurrence can be anticipated well ahead of time. Proactive action may be taken to prevent it from occurring, or to mitigate its damage. Sample risks are dependency of completion of coding done by sub-contractors, or capability of testing tools. Non Functional Tests: Non-functional testing is the testing of a software application for its non-functional requirements. Non-functional testing includes: Documentation testing Endurance testing Load testing Localization testing and Internationalization testing Performance testing

10 Security testing Scalability testing Stress testing Usability testing Volume testing Accessibility testing As quality as concerned, important non-functional tests should be planned. If any specific non-functional requirements are part of customer requirements or specification, then exhaustive testing should be planned and included into test scope. Test Schedule/Test Planning: A test plan should make an estimation of how long it will take to complete the testing phase. There are many requirements to complete testing phases. First, testers have to execute all test cases at least once. Furthermore, if a defect was found, the developers will need to fix the problem. The testers should then re-test the failed test case until it is functioning correctly. Last but not the least, the tester need to conduct regression testing towards the end of the cycle to make sure the developers did not accidentally break parts of the software while fixing another part. This can occur on test cases that were previously functioning properly. Example:

11 Exploratory testing : All testers perform ET testing for weekly 2 hrs Plan to cover all important test cases with at least 2 ET tours in 8 hrs Test Day : Dedicated 4 test days planned within MIT At least 30 to 40% code coverage(scov) Quality Gates/Quality Checks: Quality Gates can be introduced for every phases of development. Such as Design to Development, Development to Testing, etc. As part of quality gates, quality checks will be performed. Example : Development to Testing Development Completion No static check errors Unit Testing Completion Non Functional Tests coverage as part of Unit Testing Demo of complete application Code Review by internal and external architects Unit test results documented and no open issues MIT Strategy creation and roll-out Test system readiness Priority and Severity Definition as part of test strategy: Document and communicate SLA for Priority and Severity of messages Example: Very high: A message should be categorized with the priority "very high" if the problem has very serious consequences for normal business processes. Urgent work cannot be performed. This is generally caused by the following circumstances: A productive system is completely down. The imminent go-live or upgrade is jeopardized. The customer's core business processes are seriously affected. A workaround is not available. The message requires immediate processing because the malfunction may cause serious losses. High: A message should be categorized with the priority "high" if normal business processes are seriously affected. Necessary tasks cannot be performed. This is caused by incorrect or inoperable functions in the system that are required immediately. The message is to be processed a quickly as possible as a continuing malfunction can seriously disrupt the entire productive business flow.

12 Medium: A message should be categorized with the priority "medium" if normal business processes are affected. The problem is caused by incorrect or inoperable functions in the system. Low: A message should be categorized with the priority "low" if the problem has little or no effect on normal business processes. The problem is caused by incorrect or inoperable functions in the system that are not required daily, or are rarely used. Requirements traceability matrix (RTM) Ideally, the software must completely satisfy the set of requirements. From design, each requirement must be addressed in every single document in the software process. The documents include the HLD, LLD, source codes, unit test cases, integration test cases and the system test cases. In a requirements traceability matrix, the rows will have the requirements. The columns represent each document. Intersecting cells are marked when a document addresses a particular requirement with information related to the requirement ID in the document. Ideally, if every requirement is addressed in every single document, all the individual cells have valid section ids or names filled in. Then we know that every requirement is addressed. If any cells are empty, it represents that a requirement has not been correctly addressed. Test Status Collections and Reporting When test cases are executed, the test manager and the project manager must know where exactly the project stands in terms of testing activities. To know where the project stands, the inputs from the individual testers must come to the test manager. This will include, what test cases are executed, how long it took, how many test cases passed, how many failed, and how many are not executable. Also, how often the project collects the status is to be clearly stated. Some projects will have a practice of collecting the status on a daily basis or weekly basis. Test Entry and Exit Criteria Definition It is very essential to define Test Entry Creation to know when to start testing and Exit Criteria to know when to stop testing. Entry Criteria: Test system readiness, Unit test completion, no static check errors, test data availability Exit Criteria: No P1/P2 issues open, etc Test Report/Summary The management may like to have test summary on a weekly or monthly basis. If the project is very critical, they may need it even on daily basis. This section must address what kind of test summary reports will be produced for the senior management along with the frequency. The test strategy must give a clear vision of what the testing team will do for the whole project for the entire duration. This document can be presented to the client, if needed. The person, who prepares this document, must be functionally strong in the product domain, with very good experience, as this is the document that is going to drive the entire team for the testing activities. Test strategy must be clearly explained to the testing team members right at the beginning of the project.

13 References: scn.sap.com