SOFTWARE TESTING UNIT 4

Transcription

1 4.1. People and organizational issues in Testing People are an organization s most important assets. The tasks of a manager are essentially people oriented. Unless there is some understanding of people, management will be unsuccessful. Software engineering is primarily a cognitive activity. Cognitive limitations effectively limit the software process. Misconception and perception about testing 1. Testing is not technically challenging This argument is true about twenty years ago when most of the testing was manual and the products were somewhere simplistic. Normally development engineers tend to focus on specific modules. But, now testers require a holistic understanding of the entire product rather than just a single module. Suppose if the product is more complex, testing requires deep understanding of multiple domains. In the early 1990s, most people sought to be expert in C, which moved to C++ and then to java. Similarly, interms of platforms, shifted from mainframe to client server networking to network centric computing. About ten years ago, people in the testing functions found very little parallel to these kind of resume enriching. In fact, now a day the use of testing tools requires programming in languages remarkably similar to those that developers use. And also the automation tests can prove to be more challenging than even developing the code for the product. Testing is a destructive process (in the sense of finding defects in a product), there are more opportunities for out-of-box thinking. In view of all of the above, testing offers sufficient technical challenges for an interested professional and we can say there is testing in all development and development in all testing. 2. Testing does not provide me a career path or growth Testing is not a devil and development is not an angel: opportunities abound equally in testing and development. 3. I am put in testing what is wrong with me? Filling up positions for testing should not be treated as a second string function. People are sometimes made to feel that they are in testing because they could not fit in anywhere else. But, if a person is not suitable for development, for the same or similar reason: he or she may not be suitable for testing. A person should be assigned to testing only when he or she has the right aptitude and attitude for testing. Prepared by Dr. R. Kavitha Page 1

2 Appropriate recognition should be given to the Engineers who participate in activities such as testing, maintenance and documentation. 4. These folk are my adversaries The main function of testing is to find defects in the products; it is easy for an adversary attitude between the testing team and the development team. So, testing and developing teams should reinforce each other. 5. Testing is what I can do in the end if I get time Testing is not an end activity; it happens throughout and continues even after the release. 6. Testing is only destructive Test engineers not only say something doesn t work, they also point out what works in the product. The job of test engineers is not only to find defects but also prevent defects by proactively reviewing this specifications, design, architecture and code. Successful people are those who take initiative, volunteer, experiment and share their learning with others. Testing people should be able to respond quickly to changing priorities, delayed handover from development. Role of Education System The perception and misconceptions cannot be corrected by each organization individually. Higher level actions are needed and these actions pertain to the entire ecosystem covering the education system, senior management and the community as a whole. The education system does not place sufficient emphasis on testing. Here are the facts about what prevails in most universities. There are formal core courses on programming but few universities offer core courses on software testing. There are lab courses for various development tools but none or very few for common testing tools. More projects done as a part of the curriculum never ask for neither test plans nor looking at testing effectiveness. The complete weightage is on coding. Most courses and projects reward individual performance and present very little opportunity for team work, which is essential for the success of development and test engineers. Role of Senior Management Prepared by Dr. R. Kavitha Page 2

3 The senior management of an organization plays a vital role in the development of test professionals. It is simply not enough to use words like quality is our top concern or people are our top priority. This commitment has to be translated into visible actions. It is the responsibility of the management for recognition and giving reward for the top performers including testers. Role of community Test Engineers need not be just followers of technology they can take an active part in shaping and using new technology. Sometimes test engineers are under pressure to learn the technology and test the product at the same time. Learning the technology and testing the product at the same time may lead to ineffective testing. This scenario must change. Test Engineers must understand the new technology areas proactively before they are implemented in a product. Responsibilities of a senior test engineer Following the test processes for executing tests, maintaining tests and so on Filing high quality defects Categorizing the defects Adhering to schedules specified Developing high quality documentation Generation of metrics related to testing Responsibilities of a Test Lead Review of test cases, test design Planning a test strategy and test plan Allocating tasks to individuals Monitoring tasks given to individuals Making technology and tool choices for module testing Interacting with development team for debugging Analysis of metrics Overall responsibility for test quality at a module level ***** 4.2. Organization structures for Testing Teams A number of organizational structures are possible, each having its own positives and negatives. Figure shows a structure often used by small (fewer than 10) project teams. In this structure, the test group reports into the Development Manager, the person managing the work of the programmers. Prepared by Dr. R. Kavitha Page 3

4 The Development Manager's goal is to have his team develop software. Testers reporting bugs just hinder that process. Testers doing their job well on one side make the programmers look bad on the other. If the manager gives more resources and funding to the testers, they'll probably find more bugs, but the more bugs they find, the more they'll crimp the manager's goals of making software. Despite these negatives, this structure can work well if the development manager is very experienced and realizes that his goal isn't just to create software, but to create quality software. Such a manager would value the testers as equals to the programmers. This is also a very good organization for communications flow. There are minimal layers of management and the testers and programmers can very efficiently work together. Figure shows another common organizational structure where both the test group and the development group report to the manager of the project. In this arrangement, the test group often has its own lead or manager whose interest and attention is focused on the test team and their work. This independence is a great advantage when critical decisions are made regarding the software's quality. The test team's voice is equal to the voices of the programmers and other groups contributing to the product. Prepared by Dr. R. Kavitha Page 4

5 In this structure, the test group that reports to executive management has the most independence, the most authority and the most responsibility. The independence that this group holds allows them to set standards and guidelines, measure the results, and adopt processes that span multiple projects. A corporate quality standard that works well on database software might not work well when applied to a computer game. To be effective, this independent quality organization must find ways to work with all the projects they deal with and temper their enthusiasm for quality with the practicality of releasing software. In software development and testing, one size doesn't necessarily fit all, and what works for one team may not work for another. There are, however, some common metrics that can be used to measure, and guidelines that can be followed, that have been proven to work across different projects and teams for improving their quality levels. ***** 4.3. Testing Services Testing and Deubgging Goals and Polices A Goal can be described as: A statement of an accomplishment that an individual or an organization wants to achieve. Goal statement Individual/ group/ organization wants to make improvements. At the top level are General organizational goals - Ex: Goal of VCET Individual projects have specific goals. It usually reflects organizational goals. Personal Goals Each individual in an organization has a set of goals for self improvement. So that they can effectively contribute to the project. Goal statements can express expectations in quantitative terms or be more general in nature. Example: 100% testing activities are planned. (Quantitative) Prepared by Dr. R. Kavitha Page 5

6 The degree of automation of regression testing is increased from 50% to 75% over the next 3 years. (Quantitative) Testing activities are performed by a dedicated testing group. Testing group members have at least a bachelor- level degree and have taken a formal course in software testing. Note: Quantitative goals are measurable. It s useful to evaluate progress towards achieving the goal. In testing domain, goal statement should provide a high level vision of what testing is to accomplish in the organization with respect to quality of process and product. In addition to general testing goal statements, lower level goal statement should be developed for all levels of testing. Policy provides the vision and framework for decision making. It should be properly documented and it should be available for all interested parties. Organization website Disseminate the goals in website A policy statement should be formulated by a team consisting of upper management, executive personnel and technical staff. Testing policy statements reflect, integrate and support achievements of testing goals. Which will be useful for increasing the quality and customer satisfaction. Test policies also provide high level guidance as to how testing is to be done in the organization, how its effectiveness will be evaluated, who will be responsible and what choices of resources are possible. Example: How to test, what to test and who will test. Policies are not written in stone- If organization grows in maturity its policies will change. EXAMPLE: Testing policy: Our organization (Company Name) realizes that testing is an important component of the software development process and has a high impact on software quality and the degree of customer satisfaction. Delivering software of the highest quality is our company goal. The presence of defects has a negative impact on software quality. Defect affects the correctness, reliability and usability of a software product, thus rendering unsatisfactory to the client. Testing activities include traditional execution of the developing software, as well as reviews of the software deliverables produced at all stages of the life cycle. The aggregation of all testing activities performed in a systematic manner supported by organization policies, procedures and standards constitutes the testing process. A set of testing standards must be available to all interested parties on an intra organizational website. Prepared by Dr. R. Kavitha Page 6

7 Standards contain test related documents, prescribed templates, methods, procedures and tools. ***** 4.4. Test Planning A test plan is a detailed document that outlines the test strategy, Testing objectives, resources (manpower, software, hardware) required for testing, test schedule, Test Estimation and test deliverables. The test plan serves as a blueprint to conduct software testing activities as a defined process which is minutely monitored and controlled by the test manager. Making Test Plan has multiple benefits Test Plan helps us determine the effort needed to validate the quality of the application under test. Help people outside the test team such as developers, business managers, customers understand the details of testing. Prepared by Dr. R. Kavitha Page 7

8 Test Plan guides our thinking. It is like a rule book, which needs to be followed. Important aspects like test estimation, test scope, Test Strategy are documented in Test Plan, so it can be reviewed by Management Team and re-used for other projects. Making a Test Plan is the most important task of Test Management Process. Follow the seven steps below to create a test plan as per IEEE 829 Analyze the product Design the Test Strategy Define the Test Objectives Define Test Criteria Resource Planning Plan Test Environment Schedule & Estimation Determine Test Deliverables Steps for creating Test Plan 1. Analyze the Product How can you test a product without any information about it? The answer is Impossible. You must learn a product thoroughly before testing it. The product under test is Guru99 banking website. You should research clients and the end users to know their needs and expectations from the application Who will use the website? What is it used for? How will it work? What are software/ hardware the product uses? Prepared by Dr. R. Kavitha Page 8

9 2. Develop Test Strategy Test Strategy is a critical step in making a Test Plan. A Test Strategy document is a highlevel document, which is usually developed by Test Manager. This document defines: The project s testing objectives and the means to achieve them Determines testing effort and costs Back to your project, you need to develop Test Strategy for testing that banking website. You should follow steps below: 2.1. Define scope of testing 1. Before the start of any test activity, scope of the testing should be known. You must think hard about it. 2. The components of the system to be tested (hardware, software, middleware, etc.) are defined as "in scope" 3. The components of the system that will not be tested also need to be clearly defined as being "out of scope." 4. Defining the scope of your testing project is very important for all stakeholders. A precise scope helps you Give everyone a confidence & accurate information of the testing you are doing. 5. All project members will have a clear understanding about what is tested and what is not? How do you determine scope your project? To determine scope, you must Precise customer requirement Project Budget Product Specification Skills & talent of your test team Now should clearly define the "in scope" and "out of scope" of the testing. Prepared by Dr. R. Kavitha Page 9

10 As the software requirement specification, the project Guru99 Bank only focus on testing all the functions and external interface of website Guru99 Bank (in scope testing) Nonfunctional testing such as stress, performance or logical database currently will not be tested. (out of scope) Problem Scenario The customer wants you to test his API. But the project budget does not permit to do so. In such a case what will you do? Well, in such case you need to convince the customer that API testing is extra work and will consume significant resources. Give him data supporting your facts. Tell him if API testing is included in-scope the budget will increase by XYZ amount. The customer agrees and accordingly the new scopes, out of scope items are In-scope items: Functional Testing, API Testing Out of scope items: Database Testing, hardware & any other external interfaces 2.2. Identifying Testing Type A Testing Type is a standard test procedure that gives an expected test outcome. Each testing type is formulated to identify a specific type of product bugs. But, all Testing Types are aimed at achieving one common goal Early detection of all the defects before releasing the product to the customer The commonly used testing types are described as following figure Prepared by Dr. R. Kavitha Page 10

11 There are tons of Testing Types for testing software product. Your team cannot have enough efforts to handle all kind of testing. As Test Manager, you must set priority of the Testing Types: Which Testing Types should be focused for web application testing? Which Testing Types should be ignored for saving cost? 2.3. Documents risks and issues Risk is future s uncertain event with a probability of occurrence and a potential for loss. When the risk actually happens, it becomes the issue. Prepared by Dr. R. Kavitha Page 11

12 2.4.Create Test Logistics In Test Logistics, the Test Manager should answer the following questions: Who will test? When will the test occur? Who will test? You may not know exact names of the tester who will test, but the type of tester can be defined. To select the right member for specified task, you have to consider if his skill is qualified for the task or not, also estimate the project budget. Selecting wrong member for the task may cause the project to fail or delay. Person having the following skills is most ideal for performing software testing: Ability to understand customers point of view Strong desire for quality Attention to detail Good cooperation When will the test occur? Test activities must be matched with associated development activities. You will start to test when you have all required items shown in following figure: 3. Test Objectives Test Objective is the overall goal and achievement of the test execution. The objective of the testing is finding as many software defects as possible; ensure that the software under test is bug free before release. To define the test objectives, you should do the following steps: 1. List all the software features (functionality, performance, GUI ) which may need to test. 2. Define the target or the goal of the test based on above features. For Example for banking sector Prepared by Dr. R. Kavitha Page 12

13 Based on above features, you can define the Test Objective of the project Guru99 as following: Check that whether website Guru99 functionality (Account, Deposit ) is working as expected without any error or bugs in real business environment Check that the external interface of the website such as UI is working as expected and & meet the customer need. Verify the usability of the website. Are those functionalities convenient for user or not? 4. Define Test criteria Test Criteria is a standard or rule on which a test procedure or test judgment can be based. There re 2 types of test criteria as following Suspension Criteria Specify the critical suspension criteria for a test. If the suspension criteria are met during testing, the active test cycle will be suspended until the criteria are resolved. Example: If your team members report that there are 40% of test cases failed, you should suspend testing until the development team fixes all the failed cases. Prepared by Dr. R. Kavitha Page 13

14 Exit Criteria It specifies the criteria that denote a successful completion of a test phase. The exit criteria are the targeted results of the test and are necessary before proceeding to the next phase of development. Example: 95% of all critical test cases must pass. Some methods of defining exit criteria are by specifying a targeted run rate and pass rate. Run rate is ratio between number test cases executed/total test cases of test specification. For example, the test specification has total 120 TCs, but the tester only executed 100 TCs, So the run rate is 100/120 = 0.83 (83%) Pass rate is ratio between numbers test cases passed / test cases executed. For example, in above 100 TCs executed, there re 80 TCs that passed, so the pass rate is 80/100 = 0.8 (80%). This data can be retrieved in Test Metric documents. Run rate is mandatory to be 100% unless a clear reason is given. Pass rate is dependent on project scope, but achieving high pass rate is a goal. Example: Your Team has already done the test executions. They report the test result to you, and they want you to confirm the Exit Criteria. Prepared by Dr. R. Kavitha Page 14

15 In above case, the Run rate is mandatory is 100%, but the test team only completed 90% of test cases. It means the Run rate is not satisfied, so do NOT confirm the Exit Criteria 5. Resource Planning Resource plan is a detailed summary of all types of resources required to complete project task. Resource could be human, equipment and materials needed to complete a project 6. Plan Test Environment Test Environment A testing environment is a setup of software and hardware on which the testing team is going to execute test cases. The test environment consists of real business and user environment, as well as physical environments, such as server, front end running environment. 7. Schedule & Estimation Test estimation technique is to estimate the effort to complete the project. In the Test Estimation phase, the whole project has broken into small tasks and adds the estimation for each task as below Task Members Estimate effort Create the test specification Test Designer 170 man-hour Perform Test Execution Tester, Test Administrator 80 man-hour Test Report Tester 10 man-hour Test Delivery Total 20 man-hour 280 man-hour Schedule 8. Test Deliverables Prepared by Dr. R. Kavitha Page 15

16 Test Deliverables is a list of all the documents, tools and other components that has to be developed and maintained in support of the testing effort. There are different test deliverables at every phase of the software development lifecycle. Test deliverables are provided before testing phase. Test plans document. Test cases documents Test Design specifications. Test deliverables are provided during the testing Test Scripts Simulators. Test Data Test Traceability Matrix Error logs and execution logs. Test deliverables are provided after the testing cycles is over. Test Results/reports Defect Report Installation/ Test procedures guidelines Release notes ***** 4.5. TEST REPORTING Testing requires constant communication between the test team and other teams (like the development team). There are two types of reports or communication that are required. 1. Test Incident reports 2. Test summary reports 1. Test Incident Reports A test incident report is a communication that happens through the testing cycle as and when defects are encountered. Every defect has a unique ID and this is used to identify the incident. The high impact test incidents (defects) are highlighted in the test summary report. 2. Test cycle Reports Prepared by Dr. R. Kavitha Page 16

17 A test cycle entails planning and running certain tests in cycles, each cycle using a different build of the product. A test cycle report, at the end of each cycle gives 1. A summary of the activities carried out during those cycle. 2. Defects that were uncovered during that cycle, based on their severity and impact. 3. Progress from the previous cycle to the current cycle interms of defects fixed. 4. Any variations observed in effort or schedule. Test Summary Report The final step in a test cycle is to recommend the suitability of a product for release. A report that summarizes the results of a test cycle is the test summary report. Types 1. Phase wise test summary which is produced at the end of every phase. 2. Final test summary reports it is also called release test report A summary report should present 1. A summary of the activities carried out during the test cycle or phase. 2. Variance of the activities carried out form the activities planned. The tests that were planned to be run but could not be run ( with reasons) Modifications to tests Additional tests that were run Difference in effort and scheduling 3. Summary of results should include Test that failed, with any root cause Severity of impact of the defects 4. Comprehensive assessment and recommendation for release should include Fit for release assessment and Recommendation of release ***** 4.6. The Role of the Three Critical Groups in Testing Planning and Test Policy Development Three groups were identified as critical players in the testing process Managers, Developers/Testers, and Users/Clients In TMM terminology they are called the three critical views (CV) Each group views the testing process from a different perspective that is related to their particular goals, needs, and requirements. The developers/testers work with client/user groups on quality-related activities and tasks that concern user-oriented needs. Prepared by Dr. R. Kavitha Page 17

18 The focus is on soliciting client/user support, consensus, and participation in activities such as requirements analysis, usability testing, and acceptance test planning. Upper Management Goals and Policy 1. Provide adequate resources and funding to form the committees 2. Support the recommendations and policies of the committee by: Distributing testing/debugging goal/policy documents to project managers, developers, and other interested staff, Appointing a permanent team to oversee compliance and policy change making. 3. Ensure that the necessary training, education, and tools to carry out defined testing/debugging goals is made available. Assign responsibilities for testing and debugging. As representatives of the technical staff developers must ensure that the policies reflect best testing practices, are implementable, receive management support, and support among technical personnel. The activities, tasks, and responsibilities for the developers/testers include: Prepared by Dr. R. Kavitha Page 18

19 Working with management to develop testing and debugging policies and goals. Participating in the teams that oversee policy compliance and change management. Familiarizing themselves with the approved set of testing/debugging goals and policies, keeping up-to-date with revisions, and making suggestions for changes when appropriate. When developing test plans, setting testing goals for each project at each level of test that reflect organizational testing goals and policies. Carrying out testing activities that are in compliance with organizational policies. The goals and policies of users and clients reflect the organizations efforts to ensure customer/client/user satisfaction. Upper management supports this goal by: Establishing an organization wide test planning committee with funding. Ensuring that the testing policy statement and quality standards support test planning with commitment of resources, tools, templates, and training. ***** 4.7. Test Policy A Test Policy is a high level document and is at the top of the hierarchy of the Test Documentation structure. The purpose of the Test Policy document is to represent the testing philosophy of the company as a whole and to provide a direction which the testing department should adhere to and follow. It should apply to both new projects and maintenance work. Setting an appropriate test policy by senior managers provides a robust framework within which testing practitioners can then operate. This will help to ensure the maximization of the strategic value inherent in every project. Contents of a Test Policy Document 1. Definition of Testing Organizations need to be clear why they are testing. This will influence the remainder of the policy document and also the detailed testing techniques that are selected by test managers at the programme and project level. Prepared by Dr. R. Kavitha Page 19

20 From the understanding of why testing is required it is possible to specify what the purpose of testing is within the organisation. Without this fundamental linkage the test effort is destined to fail. Example: ensuring the software fulfills its requirements 2. Description of the test process It is vital to establish a solid view towards the test process. We should address questions like, which phases and subtasks will the test process include. Which roles will be involved and the document structure associated with each tasks, as well as what test levels need to be considered. Example: all test plans are written in accordance with company policy 3. Test Evaluation How are we going to evaluate the results of testing, what measures will we use to ensure test effectiveness in the project? Example: effect on business of finding a fault after its release 4. Quality Level to be achieved: Which quality criteria are going to be tested and which quality level is the system required achieving prior to its release with regards to these criteria? Example: no outstanding high severity faults prior to products release 5. Approach to Test Process Improvement How often and when are we going to assess the usefulness of the current processes in place and what elements need improving and techniques that shall be used to improve the processes. Example: project review meetings to be held after project completion ***** 4.8. Introducing the test specialist Prepared by Dr. R. Kavitha Page 20

21 Software testing specialists help develop the test plan and evaluate the quality of software components. Their role is to detect major software flaws (e.g. bugs, errors, failures, breakdowns, risks) in order to fix them and ensure the performance of the developed systems. Testing specialists have a big responsibility - they inform the testing manager of all the improvements to be made. The work of an entire team of software designers and developers depends on their analysis. The staff members who possess the following necessary skills and motivation are called test specialists. Development and application of test-related standards; Participating in requirements, design, and code reviews; Test planning Test design Test execution; Test measurement; Test monitoring (tasks, schedules, and costs); Defect tracking, and maintaining the defect repository; Acquisition of test tools and equipment; Identifying and applying new testing techniques, tools, and methodologies; Mentoring and training of new test personnel; Test reporting Skills Needed by a Test Specialist Test specialist must have: Organizational and planning skills; The ability to keep track of, and pay attention to, details; The determination to discover and solve problems; The ability to work with others and be able to resolve conflicts; The ability to mentor and train others; The ability to work with users and clients; Strong written and oral communication skills; The ability to work in a variety of environments; The ability to think creatively The first three skills are necessary because testing is detail and problem oriented. In addition, testing involves policymaking, knowledge of different types of application areas, planning, and the ability to organize and monitor information, tasks, and people. Testing also requires inter actions with many other engineering professionals such as project managers, developers, analysts, process personal, and software quality assurance staff. Test professionals often interact with clients to prepare certain types of tests, for example acceptance tests. Testers also have to prepare test-related documents and make presentations. Prepared by Dr. R. Kavitha Page 21

22 Training and mentoring of new hires to the testing group is also a part of the tester s job. In addition, test specialists must be creative, imaginative, and experiment oriented. They need to be able to visualize the many ways that a software item should be tested, and make hypotheses about the different types of defects that could occur and the different ways the software could fail. On the technical level testers need to have: An education that includes an understanding of general software engineering principles, practices, and methodologies. Strong coding skills and an understanding of code structure and behavior; A good understanding of testing principles and practices; A good understanding of basic testing strategies, methods, and techniques; The ability and experience to plan, design, and execute test cases and test procedures on multiple levels (unit, integration, etc.); A knowledge of process issues Knowledge of how networks, databases, and operating systems are organized and how they work A knowledge of configuration management; A knowledge of test-related documents and the role each documents plays in the testing process; The ability to define, collects, and analyzes test-related measurements; The ability, training, and motivation to work with testing tools and equipment; A knowledge of quality issues. Testers must have knowledge of both white and black box techniques and methods and the ability to use them to design test cases. Organizations need to realize that this knowledge is a necessary prerequisite for tool use and test automation. Testers also need to understand quality measures such as reliability, maintainability, usability, and how to test for them. Finally, testers should have some knowledge of the problem domain for which the software has been written. This knowledge, for example, can help them to understand the domain vocabulary, domain operations, and domain requirements and constraints Building a Testing Group The major activities required to manage a project and a process are: Organizing, Staffing, and Directing. These apply to managing the testing process as well. Staffing activities include filling positions, assimilating new personnel, education and training, and staff evaluation. Directing includes providing leadership, building teams, facilitating communication, motivating personnel, resolving conflicts, and delegating authority. Prepared by Dr. R. Kavitha Page 22

23 Organizing includes selecting a organizational structures, creating positions, defining responsibilities, and delegating authority. Steps in forming a test group Step 1: Support from upper management to establish a test group Step 2: Forming test group Step 3: Deciding the educational and skill levels required for each testing position. Step 4: Develop Job description Step 5: Interview Process for recruiting staff ***** Prepared by Dr. R. Kavitha Page 23