Transactions on Information and Communications Technologies vol 11, 1995 WIT Press, ISSN

Transcription

1 An approach to software product testing A. Balog, R. Trifu Center for Software Testing and Quality Evaluation, Research Institute for Informatics, 8-10 Averescu Ave., Bucharest 1, RO-71316, Romania Abstract In this paper we present an integrated approach to systematic and documented software product testing applied by the Center for Software Testing and Quality Evaluation (SofTest) within the Research Institute for Informatics (ICI). Our approach is based on the principles of black-box testing and the methods we have adopted have been applied on some complex software products in the economic, administration and management domains. The testing process is composed of a sequence of phases and activities and a set of software tools which assist the different testing tasks to be achieved. The testing process and test documents used are based on the IEEE software testing standards. This testing approach was used until now only for MS Windows applications supported by the software test tools we acquired. 1 Overview The software testing process applied within the frame of our Center is consisted of a succession of phases such as test planning, set test design, execution and evaluation, each of them having principal activities and a series of tasks to be accomplished (fig.l). These have been described in Balog, lonescu & Trifu [3] and Balog & Trifu [4] previous works. Along the test planning are used classical methods of planning such as Gantt charts and PERT, the tests set designing phase is founded on some ideas inspired from McCabe's paper [6]. At the end of each activity a document is provided, the most important being the Test Plan, the Test Procedures Specifications and the Test Summary Report, according with IEEE standards [1], [2].

2 374 Software Quality Management The testing process is assisted in the first phase by planning tools (e.g. MS Project for Windows) and SQA Manager [9]. In the second phase we use a case tool for designing the DFD's and identify, with a numeric code, each function and subfunction of the software product. This decomposition is the basis for the test procedure and test cases specifications. The execution and evaluation phase is assisted by the SQA Robot tool which registers the test procedures, make a test cycle workflow tracking and reports the test procedures execution. The analysis made on the basis of the logs files generated by SQARobot, allows the accomplishment of the Test Summary Report. SQA Robot is part of the SQA TeamTest from Software Quality Automation, Inc. [8], which is an integrated environment for GUI client/server applications on IBM PC MS Windows. Figure 1: Testing Process - phases, activities, documents, and tools 2 Software testing process description 2.1 Test Planning Phase The planning of the general approach, resources, and the schedule of the testing process is developed for each software product to be tested and is recorded in the Test Plan Documentation. Activities The most important activities which are done in this phase are the

3 Software Quality Management 375 identification and specification of the product test environment, like hardware and software platforms and test tools, test data, the type of testing, such as functional, usability and performance testing. There are, also, identified the personnel requirements, the assignment of test responsibilities, the testing schedule, and there are selected the software product features to be tested. Documents At the end of this phase all the information identified along the mentioned activities are recorded in Test Plan Documentation, which has the following succession of relevant sections: test plan identifier, test items, features to be tested, testing approach test deliverables, hardware and software requirements, responsibilities, personnel needs, schedule. Tools The test planning activities are aided by two software tools. MS Project for Windows used mostly for scheduling and SQA Manager which leads plan testing activity and assemble resources to ensure a manageable, repeatable process. SQA Manager guides through the test planning process with test documents that adhere to the IEEE software testing standards. 2.2 Tests Set Designing Phase This phase is one of the most important, because all the subsequent phases are depending on it. Here are developed the test procedures and their associated test cases specifications. These are developed following several test design techniques and their mathematical foundations inspired from Beizer's books [5], and from Myers's [7] methodologies for the design of effective test cases. Activities For designing the test procedures, first of all, is applied a functional decomposition method for the software product, done after studying the product user's documentation and inspection of the software items. This method is derived from the McCabe's Structured Testing and is composed of a sequence of steps. The functional decomposition begins with the drawing of the DFD's hierarchy and uniquely labeling each process. The labeling technique is employed from the hierarchy numbering scheme used in DFD's notation, starting from the highest level to the lowest. So there are distinctively identified all the software product functions and their associated subfunctions. The following step is that of specifying the flow test by developing the test sets. Each set is a combination of codes representing a succession of functions and subfunctions to be executed having as target a final function to be accomplished. A set is one of the paths inside the functional software product graph recommended for test. From the total numbers of possible combination is chosen a number of paths to be tested. This number is determined using the cyclomatic complexity formula. After identifying the paths, each of them or a combination are becoming test procedures which are

4 376 Software Quality Management distinctly coded and there are associated with test cases, them also being uniquely and significantly coded. The final step is that of designing hierarchical diagram (fig. 2) having on the first level the main software product functions, on the second level the associated test procedures and on the third one the assigned test cases. Software product FUNCTIONS TEST PROCEDURES aa..a TEST CASES c5o...c Figure 2: Hierarchical diagram of the functions, procedures, and test cases Documents In this phase there are elaborated a set of documents like Test Design Specification, Test Procedures Specifications and Test Case Specifications. Each of them containing mandatory paragraphs (Table 1). Table 1 - Developed Documents TEST DESIGN SPECIFICATION Purpose Identifier Features to be tested Test identification Feature pass/fail criteria TEST CASE SPECIFICATION Purpose Identifier Test items Inputs spec. Outputs spec. TEST PROCEDURE SPECIFICATION Purpose Identifier Special requirements Steps: log, set up, start, proceed, shutdown, restart, stop.

5 Software Quality Management 377 Tools The software tool used in the tests set designing phase is a CASE tool which is a support for representing the hierarchy of DFD's and the labeling schema. 2.3 Measurement Phase The measurement phase is based on the test procedures and specifications developed in the previous phase. test cases Activities The measurement phase is made of an execution activity and evaluation activity. First of all there are verified the test data, the test resources, test case and procedures specifications. Then the tests are run and there are determined and logged the results. The termination of each test is checked even if it is normal or abnormal. Finally the each logged result is verified, including the pass/fail conditions of termination fulfilled or not. For the activity of evaluation the test efforts and the software product, the tasks to be made are mostly connected with the completion of Test Summary Report. So it is described the testing status, concerning the abnormal termination and identified the areas not sufficiently tested and recorded the reasons. Also is described the product status and there are recorded the differences, discovered during the testing, between the software product and its documentation. Documents The Test Summary Report has a few sections which are elaborated throughout the execution and evaluation activities. The test execution is registered in the sections of the Test Summary Report as it follows. In the Summary Results there are recorded the incidents, the pass/fail results, their explanation and in the Summary Activities are described the normal\abnormal termination and the consumption of resources. The testing evaluation is registered in the Comprehensiveness Assessment section, where are mentioned the reasons for abnormal termination. In the Variance section are described the dissimilarities between the software product and its documentation. Tools The test tool used for automated test execution is SQA Robot which records the test execution actions through the Object-Oriented Recording technology and also permits the inclusion of predefined test cases options. The tool generates the test scripts in Visual Basic, so these can be extended with more sophisticated test cases. The test procedures, so recorded, can be run any time interactively or

6 378 Software Quality Management completely unattended. The test results are registered in a set of log files for each test procedure indicating the pass or fail for each test case included. 3 Conclusions This paper described an integrated approach to software product testing. The approach have been cost effective and applied to very large products. Quality improvements were realized with organizational structure that ensures independence and objectivity in software product testing and evaluation. Benefits results from using proven product testing methods and procedures, and from an expansion of the role of the SofTest to encompass quality assurance. Our future objectives include increasing the quality of the release in the field and decreasing the product testing interval through automatization and establishment of other quality processes. References 1. ANSI/IEEE Std IEEE Standard for Software Unit Testing. 2. ANSI/IEEE Std IEEE Std. for Software Test Documentation. 3. Balog, A., lonescu, M., Trifu, R. Software testing for certification, in IE'93, pp. 136, Proceedings of the 1st International Conference on Information Economics, Bucharest, Romania, May 19-22, 1993 (in rumanian). 4. Balog, A., Trifu, R. Software product quality specification and evaluation, Technical Report, Research Institute for Informatics (ICI), 32 pp., June 1993 (in rumanian). 5. Beizer, B. Software Testing Techniques, Van Nostrand Reinhold, New York, McCabe, Th.J., Schulmeyer, G.G. System testing aided by structured analysis: a practical experience, IEEE Transactions on Software Engineering, vol.se-11, no.9, Sept. 1985, pp Myers, G.J. The Art of Software Testing, John Wiley&Sons, New York, SQA Robot User Guide, 1993, Software Quality Automation, Inc. 9. SQA Manager User Guide, 1992, Software Quality Automation, Inc.