Arcade Game Maker Product Line System Test Plan Template

Transcription

1 Arcade Game Maker Product Line System Test Plan Template ArcadeGame Team July 2003

2

3 Table of Contents Overview Error! Bookmark not defined. 1.1 Identification Error! Bookmark not defined. 1.2 Document MapError! Bookmark not defined. 1.3 Concepts Error! Bookmark not defined. 1.4 Reusable ComponentsError! Bookmark not defined. 1.5 Product line variationerror! Bookmark not defined. 1.6 Readership Error! Bookmark not defined. 2 Product Line ContextError! Bookmark not defined. 2.1 In Scope Error! Bookmark not defined. 2.2 Out of scope Error! Bookmark not defined. 3 Approach Error! Bookmark not defined. 4 References 1 CMU/SEI-2001-TR-001 i

4

5 1 Overview 1.1 Identification The Arcade Game Maker Product Line organization will produce a series of arcade games ranging from low obstacle count to high with a range of interaction effects. More detailed information can be found in the Arcade Game Maker scope document. This document describes how a product is tested in the AGM product line. 1.2 Document Map The Arcade Game Maker Product Line is described in a series of documents. These documents are related to each other as shown in Figure 1. This map shows the order in which the documents should be read for the first time. Once the reader is familiar with the documents, the reader can go directly to the information needed. This is the system test plan template. Product Line organizations use this document to capture how the product teams test a new product. This document follows the outline provided in [McGregor 01b]. 1

6 Figure 1- Document Map 1.3 Concepts A software product line is a group of products sharing a common, managed set of features that satisfy specific needs of a selected market or mission and that are developed from a common set of core assets in a prescribed way. A product line core asset is any deliverable that is intended for continuing use in multiple products. A product is any saleable or useable output produced from the product line assets. Application engineering is the construction of a product using the product line architecture and selecting from the inventory of available product line components. It is accomplished by identifying the specific values for each variation point in the product line architecture. The organization that manages this set of activities is responsible for using the assets to produce a single product. This team is responsible for achieving the specific qualities defined for the product. Domain engineering creates, acquires, and manages the resources needed to build an entire set of products. The organization that manages this set of activities is responsible for long- 2 CMU/SEI-2001-TR-001

7 term maintenance of individual assets that will be used across products. The organization produces sufficient assets to cover the variations in the architecture. 1.4 Readership This document is intended to provide some level of information to all of the stakeholders in the Arcade Game Maker framework but is primarily intended for product development teams. The system test plan template describes for a manager the resources that are needed to produce a product. Technical members of the organization can use the system test plan template as the detailed process for testing a product. 3

8 2 Plan The AGM product line organization has decided to follow the IEEE 829 standard for test plans. The following template is filled in for each product. See [McGregor 01b] for definitions of each section. 1. Introduction 2. Test Items 3. Tested Features 4. Features Not Tested (per cycle) 5. Testing Strategy and Approach a. 5.1 Syntax b. 5.2 Description of Functionality c. 5.3 Arguments for tests d. 5.4 Expected Output e. 5.5 Specific Exclusions f. 5.6 Dependencies g. 5.7 Test Case Success/Failure Criteria 6. Pass/Fail Criteria for the Complete Test Cycle 7. Entrance Criteria/Exit Criteria 8. Test Suspension Criteria and Resumption Requirements 9. Test Deliverables/Status Communications Vehicles 10. Testing Tasks 11. Hardware and Software Requirements 12. Problem Determination and Correction Responsibilities 13. Staffing and Training Needs/Assignments 14. Test Schedules 15. Risks and Contingencies 4 CMU/SEI-2001-TR-001

9 3 Analyses In this section we describe the techniques and the agreed-upon standards used to test each product in the AGM product line. These are justified and put into context in [McGregor 01b]. 3.1 Coverage Standards The system test coverage standard is based on information in each use case. There are two fields in each use case in the AGM requirements model that are used to determine the correct level of test coverage: frequency and criticality. Both are rated on scales from high to low. Table 1 shows the combinations of these values and the coverage standard defined for each. Table 1 Coverage standards Value combination Frequency, criticality Coverage criteria Details High, high main alternative v v For each tested cenario construct test cases for all pair-wise combinations of scenario variables. exceptional v High, low main alternative exceptional v v For each tested scenario construct test cases for all pair-wise combinations of scenario variables that are defined in the system state machine. Medium, high main alternative exceptional v For each tested scenario construct test cases for some pair-wise combinations of scenario variables that are defined in the system state machine. Medium, low main alternative exceptional v For each tested scenario construct test cases for some pair-wise combinations of scenario variables that are defined in the system state machine. 3.2 Analyses The analyses needed to support the creation of test cases from the use cases are conducted by filling out the tables below. Example values are given in the tables to illustrate their use. 5

10 Table 2 Data Type Analysis Variable Data Type Equivalence classes age int 0 18; 18-21;22-65; 66- Table 3 Test case construction Test Case No. Variable 1 )age) Variable 2 Variable 3 Variable CMU/SEI-2001-TR-001

11 4 Modifying the System Test Plan A specific system test plan is modified every time the requirements for the system being tested is changed using techniques discussed in section 3.2. The generic system test plan is modified when it is shown that the techniques are not producing effective test cases and the standards are not producing satisfactory results. That is the focus of this attached process. Collect test effectiveness data End of increment? yes no Compute test effectiveness (TE) statistics TE < 75%? no yes Modify the standards and techniques Figure 2-Attached process 7

12 4.1 Test effectiveness Test effectiveness is measured by how many defects escape detection by the tests that are run. For the unit test phase, any defect found in a component after unit test counts against the test effectiveness of the unit test. It is computed as: TE? TotalDefects? DefectsFoundAfterTest TotalDefects Defects are cataloged as they are identified and analyzed to determine their origin. At the end of a product line increment the test effectiveness is computed on a component by component basis. When the average TE goes below 75% the test coverage standards are made more comprehensive. 8 CMU/SEI-2001-TR-001

13 5 References and Further Reading For references see the Bibliography document. 9

14