Defining the essential terms in testing Jokinen Tauno & Määttä Juha University of Oulu

Size: px

Start display at page:

Download "Defining the essential terms in testing Jokinen Tauno & Määttä Juha University of Oulu"

Leonard Adams
6 years ago
Views:

1 Defining the essential terms in testing Jokinen Tauno & Määttä Juha University of Oulu Abstract ing comprises several terms that are not unambiguous or uniform across the discrete disciplines. Disparate definitions prevent to obtain coherent conception of testing profession and complicate to establish acknowledged education for testing. Therefore the testing professional are also undervalued and the masses are not pursuing to testing profession.this paper examines the existent terms and definitions of testing domain intending to determine unambiguous and universal definitions, which can be used within diverse disciplines.the discussed terms do not cover the entire terminology of testing domain; rather just a few term is considered aiming to provide definitions for terms essential especially in ICT (Information and Communication Technology) industry. Our focus in definitions is on the economical value of testing, thus the risk and value based approach is applied. 1. Introduction ing comprises several terms that are not unambiguous or uniform across the discrete disciplines. Disparate definitions prevent to obtain coherent conception of testing profession and complicate to establish acknowledged education for testing. Therefore the testing professional are also undervalued and the masses are not pursuing to testing profession. Especially the term testing is problematic due to wide concept of the term. There are many other terms close to testing that may or may not be seen as testing such as terms measuring, simulating, verification, and validation. Nevertheless, the purpose of given definition for testing is not to exclude these activities rather contemporary testing encompasses all of these in a one way or another. 2. Defining testing and related terms ing as a term has not a single clear definition that is accepted widely to be used among various disciplines, thus the focus in testing is not clear and definitions do not emphasize the risk approach and risk mitigation, which are essential aspects. ing has several meanings depending on the discipline and subject where testing is conducted. In addition to the broad concept, testing is also applied for numerous subjects such as materials, components, subassemblies, applications, techniques, technologies, methods and processes. Furthermore, testing may be focused on a specific attribute within the subject such as functionality, performance, capacity, usability, reliability, security and so forth. Software testing literature has various definitions for testing whereas literature of electronics testing lacks the definitions completely (e.g. Davis 1995, Landers et al and Burns&Roberts 2001). Mostly the definitions have right dimension for testing, but usually overlook many vital aspects.

2 Consideration of the maturity of testing within organizations complicates the determination of testing. One testing maturity model defines the purpose and focus of testing in following way within each maturity level: Maturity level 1: To show the product works Maturity level 2: Defect detection Maturity level 3: Functional requirements verification and validation Maturity level 4: Quality measurement Maturity level 5: Quality control. (Ham et al. 2001). Maturity approach reveals that the current typical definitions for testing are primarily for lower maturity levels. This, in turn, addresses the fact that expectations for quality and total value of products have changed over the time. Therefore the requirements for contemporary product development and testing as well have changed accordingly. This evolution from regulative and standard based quality to customer value based quality optimization should be perceived in testing business and in definition of testing because testing faces different challenges in evaluating the total value of the products from customers point of view. This customer value based quality assessment will be the key in maintaining customer loyalty and obtaining additional market share. This approach shifts the focus of testing from combating symptoms to preventing the symptoms to appear. Besides, it enables to focus on the things that really matters to customers. The problem with this approach arises with the current definitions of testing, which emphasizes reactive attitude in testing. In addition to changes in attitudes, proactive and value based approach requires new techniques and methods for assessing new technologies, features and products. We examined variety of definitions for testing (e.g. Myers 1979, Pol&van Veenendaal 1998 and IEEE Std ), and due to absence of appropriate definition, we determined testing being as follows: ing is a process of performing planned, prepared and controlled activities to evaluate the uncertain quality level of the object or/and processes, assisting to determine whether those meet the requirements or/and the specifications set prior test. This definition has following elements essential in focusing on risk based testing: Proactive and systematic approach Feedback information Predefined requirements for test Deals with particular uncertainty. These aspects exclude all the experimental activities, such as piloting, from the testing requiring disciplined approach to testing operations. The verbal definition is somewhat rigid, but Figure 1 illustrates the operational framework of testing clarifying the definition of testing. Particular risk or uncertainty associated ing Objectives & Requirements Managing Implementation Implementation (Process) Process control data ing To be reworked Result Evaluation Detailed diagnosis Figure 1. ing in operational framework Verification and Validation Pass Fail Mitigated risk/ diminished uncertainty Outcome British Standard (BS ) defines verification as follows: Confirmation by examination and provision of objective evidence that specified requirements have been fulfilled whereas validation is defined in a following way: Confirmation by

3 examination and provision of objective evidence that the particular requirements for a specific intended use have been fulfilled. Based on these definitions, it is not anymore outlandish that these terms are used contradictorily. In the past, verification is understood as a human examination or review of the work product, which is called static testing, and validation is usually associated with traditional execution-based testing, which is called dynamic testing. Nevertheless, this is not an issue nowadays but verification can employ dynamic testing and validation employs static testing as well. To differentiate these terms, validation can be replaced by the term acceptance in order to clarify the difference between the meanings of verification and validation. By replacing the validation with acceptance it emphasizes that object is evaluated towards the original and intended requirements whereas the verification is performed to evaluate the object against the explicit specifications that is set prior verification process. ing encompasses several other controversial terms than just verification and validation (V&V) such as measuring, simulating, modeling, review, and inspection. All of these are used to perform certain type of testing, but not all measuring, neither the others, can be defined as testing activity due to varying level of associated uncertainty. Therefore the Figure 2 is presented to illustrate some dimensions of testing in terms of used activities. In an extensive meaning of the term testing all the presented activities are involved, but testing as a profession cannot be applied to all measuring, simulating, modeling, review, and inspection activities. For instance, measuring can be applied to gather information about the critical properties of the certain component in order to establish criterion for testing, but the measuring activity is not testing in this case, however. When the measuring is performed to diminish the uncertainty whether or not the component meets the established requirements it is a testing activity. Simulating Simulating Measuring Measuring Dimensions of ing V&V V&V Implement Modeling Modeling ing ing Implement Reviews Reviews Inspections Inspections Figure 2. Activities involved in testing function. And other activities Quality assurance comprises V&V activities completely, thus V&V could be used instead of the term testing to embody the activities involved in defect prevention, defect detection, and quality control. Nevertheless, the various enterprises have testing organizations and testing teams instead of V&V functions, thus the term testing stands for the activities employed by V&V in defect prevention, defect detection, and quality control ing performance measures ing performance measures are essential in managing, controlling, and improving testing activities. Nowadays business requires clearly defined measures in order to allow objective financial judging and assessment about the performance of various functions within organizations. However, testing performance measures, such as efficiency and effectiveness of testing are rather structural and technical performance measures instead of business based financial measures. Therefore the intention is to justify and define business based performance measures in this study. Figure 3 clarifies the aimed relationship of test efficiency, test effectiveness, and

4 economic test coverage which are defined below. Indicates how much recources have been spent in relation to achieved coverage Efficiency Efficiency Effectiveness Effectiveness Coverage Coverage Figure 3 Relationship of the test efficiency, effectiveness and coverage Effectiveness Indicates how well the desired coverage is ahcieved Effectiveness of testing measures the quality of efforts used for testing. Effectiveness indicates how well the desired test coverage has been achieved. According to Black (2004), effectiveness means producing desirable and valuable results. In order to produce desirable and valuable results, those have to be known. Black (2004) has listed four ways testers produce desirable and valuable results: 1. To find bugs that get fixed or even prevent bugs. 2. To find bugs that don t get fixed but are known. 3. To run tests mitigating risks. 4. To help guide the project with timely, accurate, and credible information. Kaner et al. (2002) uses following analogy cited from The Effective Executive by Peter Drucker to explain effectiveness: Executives are always given more to do than they can possibly achieve. Effective ones pick a subset of tasks that they will do well and skip many other tasks entirely. Ineffective executives try to do everything, don t succeed, and don t do particularly well at much of what they did attempt. valuable outcomes of testing listed previously. To obtain effective testing is a challenging task because in any real-world complex system, the number of tests that could be conceivably run would take person-years or person-centuries to develop and execute (Black 2004). Thereby, effectiveness of testing is not merely a measure of the obtained coverage of the tested product, but also a measure of the coverage of critical and essential factors based on the risks related to overall business, the development project, and the product. Later specified definition of test coverage is determined to be divergent in this study to emphasize what is the effectiveness of testing. This definition encourages applying Pareto rule to find out the critical 20 % of test cases covering the 80 % of the most significant risks related to development project Universe of Cases As stated previously, testing comprises unlimited amount of conceivable test cases. This is called also as a universe of test cases. To point out the reality of possible test cases, Davis (1997) presents an example of a simple digital device with 64 input pins. To test every possible combination of ones and zeros and monitoring the results of the tests on the output pins at the one hundred million tests per second rate this will take 5850 years. Actual amount of test cases is 2 64 yielding about eighteen million million million. Really simple software could have millions of possible test paths as well. Basic systems consisting of both hardware and software would then comprise unlimited universe of possible test cases. To understand what is effective testing, presented analogy has to be understood and put it to the context of four desirable and

5 Coverage Based on the space of conceivable test cases, test coverage would be always zero percent because of unlimited amount of test cases. Generating tests for all cases is infeasible. Though, test coverage is traditionally defined as a degree to which a given test or a set of tests addresses all specified requirements for a given system or component (IEEE Std ). This definition assumes that specified requirements are set for testing in terms of quality and reliability. Figure 4 illustrates the relation between space of test cases and traditional test coverage. Unlimited space of possible test cases Defined test cases Selected test cases ( coverage) Figure 4. Relation between the space of test cases and traditional test coverage. Black (2004) has divided the definition of test coverage to two category; behavioral and structural. Behavioral coverage assessments, such as those that look at requirements coverage, are related most closely with business risk. ing should cover each area proportional to the impact of failure on the business. In contrast, to assess the coverage of technical risks, structural design elements are considered. In structural coverage assessment identifying the major software and hardware subsystems in a larger system and the data and control flows between them provides a model for assessing technical test coverage. This partitioning can be made further up to individual lines of source code to look at technical risk in terms of complexity. Literature considering electronics testing refers often to term fault coverage. Fault coverage is a measure of the ability of a tester to detect defects (Davis 1997). Nonetheless, fault coverage is an indefinite indicator because the all possible defects are not known. Of course there can be defined certain defects and compare some tester against those defects, but still we don t know actual fault coverage. Basically, fault coverage seems to be used in comparing rival test systems to figure out the performance of each one. In addition, fault coverage as term is used just in the literature concentrating testing of electronics. There is also conflicting use of the terms test coverage and fault coverage in the literature where these have been used as equal terms (e.g. Burns&Roberts 2001). Thus, term fault coverage should be discarded in order to avoid misunderstandings and inconsistencies in the use of terms. In this paper, the term test coverage is defined differently compared for those traditional definitions discussed above. The aim of this paper is to emphasize the value of testing, thus the terms should emphasize this approach as well. Thereby the test coverage is defined as measure of the economic coverage of the most significant risks obtained in product process. This, in turn, means that the test coverage in this paper focuses on the financial impact of testing as depicted in Figure 5 below. The defined test coverage requires that risks are assessed and measured financially and recorded on economic value basis. coverage The benefits of testing outweigh the costs of testing The benefits obtained in covering these risks do not equate or exceed the costs of testing Figure 5. Economic test coverage based on business risk.

6 Efficiency efficiency is a measure of the use of resources in testing, that is, how much effort it took to find the desired and valuable results mentioned previously. Black (2004) emphasizes that resources include not only money, such as salaries, tools, and host systems, but also time. Efficiency of testing is an important indicator in terms of financial measurement. Efficiency is highly dependant of the effectiveness of testing. By doing right testing, on right time, in right phase testing will be effective. Respectively, efficiency can be improved within that chain of right procedures supplemented with right techniques, methods, and tools enabling efficient use of resources and rational test processes. 3. Conclusions This paper proved the inconsistency associated with various terms in testing domain. This paper provided insight into testing profession through the essential terms related to testing. Definitions for several terms were examined and if necessary, the terms were determined in this paper. The risk approach was incorporated into the defined terms in this paper intending to emphasize the economical value of testing instead of merely encompassing testing and related terms as defect detection activity. The defined terms highlight the value and significance of testing operations, therefore those increase the interests, appreciation, and understanding of testing domain. This influnces in positive way for enhancing the expertise within the testing domain as well as unambigious definitions contribute to obtain authentic conception of testing profession. References Black, R Critical testing process - Plan, Prepare, Perform, Perfect. Boston, Addison-Wesley. 571 pp. BS Vocabulary of terms in software testing. British Computer Society Specialist Interest Group in Software ing (BCS SIGIST). Burns, M.& Roberts, G.W An Introduction to Mixed-Signal IC and Measurement. New York, Oxford University Press. 684 pp. Davis, B The economics of automatic testing. London, McGraw-Hill Book Company. 408 pp. Ham, M., Jacobs, J., Swinkels, R.& van Veenendaal, E Metric Based ing Maturity Model Framwork v1.1. [WWWdocument]. < Read IEEE Std IEEE Standard Glossary of Software Engineering Terminology. The Institute of Electrical and Electronics Engineers, 83 pp. Kaner, C., Bach, J.& Pettichord, B Lessons learned in software testing. New York, John Wiley & Sons Inc. 286 pp. Landers, T.L., Brown, W.D., Fant, E.W., Malstrom, E.M.& Schmitt, N.M Electronics manufacturing process. Englewood Cliffs, Prentice Hall. 564 pp. Myers, G. J The Art of Software ing. New York, John Wiley & Sons Inc. 177 pp.

7 Pol, M.& Veenendaal, E Structured testing of information systems an introduction to TMap. Deventer, Kluwer BedrijfsInformatie. 154 pp.

Testing. Testing is the most important component of software development that must be performed throughout the life cycle

Testing. Testing is the most important component of software development that must be performed throughout the life cycle Testing Testing is the most important component of software development that must be performed throughout the life cycle Testing must be carried out by developers continuously More methodical testing must