0 Introduction Test strategy A Test Strategy for single high-level test B Combined testing strategy for high-level tests...

Transcription

1 TPI Automotive Test Process Improvement Version: 1.01 Author: Sogeti Deutschland GmbH Datum: Sogeti Deutschland GmbH. Version

2 0 Introduction Test strategy A Test Strategy for single high-level test B Combined testing strategy for high-level tests C Combined strategy for high-level tests plus low-level tests or evaluation D Combined strategy for all test and evaluation levels Life cycle model A Planning, Design, Execution B Planning, Preparation, Design, Execution and Completion Moment of involvement A Completion of test basis B Start of test basis C Start of requirements definition D Project initiation Estimating and planning A Substantiated estimating and planning B Statistically substantiated estimating and planning Test design techniques A Informal techniques B Formal techniques C Mathematical methods Static test techniques A Inspection of test basis B Checklists Metrics A Project metrics (product) B Project metrics (process) C System metrics D Organization metrics (>1 system) Test automation A Use of tools B Managed test automation C Optimal test automation Test Environment...48 Sogeti Deutschland GmbH. Version

3 9.A Managed and controlled test environment B Testing in the most suitable environment C Environment on call Office and laboratory environment A Adequate and timely office and laboratory environment Commitment and motivation A Assignment of budget and time B Testing integrated in project organization C Test engineering Test functions and training A Test manager, integrator and testers B (Formal) Methodical, Technical and Functional support, Management of the test process, testware and infrastructure C Formal internal reviewing Scope of methodology A Project specific B Project specific with external scope C Organization generic D Organization optimizing, R&D activities Communication A Internal communication B Project communication (defects, change control) C Communication in organization about the quality of the test processes Reporting A Defects B Progress (status of tests and products), activities (costs and time, milestones), defects with priorities C Risks and recommendations, substantiated with metrics D Recommendations have a Software Process Improvement character Defect management A Internal defect management B Extensive defect management with flexible reporting facilities C Project defect management Testware management A Internal testware management...78 Sogeti Deutschland GmbH. Version

4 17.B External management of test basis and test object and traceability of system requirements to test cases C Reusable testware Test process management A Planning and execution B Planning, executing, monitoring and adjusting C Monitoring and adjustment in organization Evaluation A Informal evaluation B Evaluation techniques C Evaluation strategy Low-level testing A Life cycle: planning, design and execution B White-box techniques C Low-level test strategy Integration testing A Integration identified as a separate and planned process B Test strategy for integration C Standardized approach for integration Test maturity matrix Overview dependencies Glossary Literature Sogeti Deutschland GmbH. Version

5 0 INTRODUCTION In modern car electronics and software development, testing is an important part of the development process. The growing complexity of these electronics and software demands the enhancement of the development process and therefore also of the test process. Since 1998, TPI (Test Process Improvement) [Koomen and Pol, 1999] offers an instrument to stepwise improve the test process. Since then, it was introduced in a lot of projects to bring the test process to a higher level. Initiated by the German automotive industry, the development of TPI Automotive was started in 2003 (and finished in 2004). The objective was to create a model with all the ingredients of the existing TPI-model but translated and extended to cover the automotive peculiarities. The TPI model offers a framework to determine the strengths and weaknesses of the current test process within a project or organization. On the other hand, the model offers support in the formulation of adequate and realizable improvement suggestions for the test process in terms of quality, cost and time. In this introduction a short description is given of the model. This description is followed by an explanation of how the terms high-level, low-level and integration testing are used in this model and what that means for the usage of the model. of the model The TPI model can be visualized as follows: Key Areas Levels TPI Matrix Checkpoints Improvement Suggestions Key Areas In the TPI Automotive-model 21 Key Areas are used covering all the aspects of a structured test process. One key area identifies a typical aspect of the test process and offers a guideline how to improve the test process related to this aspect. Key Area Key Area Test strategy Test functions and training Life-cycle model Scope of methodology Moment of involvement Communication Estimation and planning Reporting Test design techniques Defect management Static test techniques Testware management Metrics Test process management Test automation Evaluation Test environment Low-level testing Office and laboratory environment Integration testing Commitment and motivation Sogeti Deutschland GmbH. Version

6 Levels For every key area different maturity levels are found, starting with Level A for all key areas up to level D for some key areas. Every next level differs in his nature of maturity from the previous one. Every higher maturity level implies the lower maturity level. With determining the maturity level for every key area the current situation of the test process is visualized. Checkpoints To make an objective determination whether the maturity level applies to a certain key area, Checkpoints are given. Only when all the checkpoints belonging to a maturity level are fulfilled, this level applies. Test Maturity Matrix (TPI Matrix) The key areas and their maturity levels are combined in the Test Maturity Matrix (TPI Matrix; see Chapter 22). The distribution of the maturity levels is based on priorities and dependencies between the maturity levels of different key areas. For example to start with test automation, it is necessary to have test cases. These test cases can be derived by using test design techniques. Therefore the maturity level A of the key area Test design techniques has a higher priority (meaning it is placed more to the left in the matrix) as maturity level A of the key area Test automation. Maturity level A of the key area Metrics depends on level B of Test process management, level A of Defect management, Level B Reporting and level B of Commitment and Motivation. Level A of the key area Metrics is therefore situated as far to the right as level B of the key area Commitment and motivation. (All the dependencies between the different levels of the key areas are mentioned in chapter 23). Before improvement suggestions can be formulated, it is necessary to determine the maturity level of each single key area. An analysis of the test process is carried out to collect all necessary information to determine whether the checkpoints of the key areas are fulfilled. If the checkpoints are met, the maturity level applies to the key area. The results are visualized within the matrix, showing which maturity levels apply to which key areas. The same matrix can be used to visualize which maturity levels for the different key areas are desired. Indications, to which measures should be taken to bring the test process to these desired levels, are mentioned in the form of improvement suggestions. Improvement suggestions In the model for every maturity level of each key area, Improvement Suggestions are given. These improvement suggestions can be used as a basis to define tailor made (dedicated to the process that is analyzed) improvement activities. The implementation of these improvement activities should bring the organization the desired maturity levels for the respective key areas. Application of the TPI Automotive model Every change process follows more or less the same procedure: changes are implemented on the basis of specific objectives in order to improve the process from the current towards the desired maturity. The improvement of the test process doesn t differ here from other change processes. The following figure shows the Sogeti Deutschland GmbH. Version

7 different activities of a change process for testing. These activities are described, with special emphasis on the items where the TPI Automotive model can play a specific role. Obtain awareness Determine target, area of consideration, and approach Execute assignment Define Improvement actions Perform evaluation Formulate plan Implement improvement action Obtain awareness The first activity in the change process is to obtain awareness for the necessity to improve the test process. The reason to improve the test process usually lies in experiencing a number of problems within or surrounding testing. The improvement of the test process is usually seen as one of the possibilities to solve these problems. This awareness also implies an agreement on main issues by the parties involved as well as their commitment to the change process. It isn t enough when there is commitment at the start of the change process, this commitment should be maintained throughout the process. This requires a dedicated and continuous effort. Determine objectives, scope and approach This activity includes the formulation of objectives, scope and approach for the change process. Should testing become cheaper, faster or better? Which test levels and which test projects are considered, how much time is available for the improvement activities and which cost are estimated? Execute analysis The analysis is executed in order to describe the current situation of the test process. The usage of the TPI Automotive model is an important part of the analysis, because it offers a framework to determine the strengths and weaknesses of the process under consideration. On the basis of collected data from interviews and document study, for each key area the levels are evaluated by means of the checkpoints. The Test Maturity Matrix is used in order to show the integral situation of the test process. The matrix clearly shows the strengths and weaknesses of the test process by coloring the obtained levels per key area in the matrix. Define improvement activities On the basis of the improvement objectives and the result of the analysis, the improvement activities are defined. The activities are bound to the rule that they support a stepwise and smooth improvement of the test process. The TPI Automotive model helps to determine the improvement activities. The levels of the key areas and the TPI matrix give several possibilities to determine improvement steps. Depending on the objectives, the scope, the timeframe and the Sogeti Deutschland GmbH. Version

8 results of the analysis, it is possible to implement improvement activities for one or more key areas. For each selected key area it is possible to choose for an improvement to the next level or on specific occasions to the subsequent level. Furthermore the TPI Automotive model offers a large amount of improvement suggestions to help achieving the next level of maturity for a specific key area. Formulate plan A plan is formulated in order to obtain commitment and engagement of all stakeholders for the activities to be implemented. The plan includes objectives and a planning of activities in order to achieve the objectives. The plan consists both of activities related to content as well as activities related to the process of change as such. Implement improvement activities The improvement activities are implemented. In this phase the hidden resistance to the improvement activities becomes visible. The implemented activities should be monitored in terms of being and to what extent they were executed. A useful instrument is the so called "self assessment", where the TPI Automotive model is used to quickly determine the current state of the test process. The people involved in the process evaluate their own process with the help of the TPI Automotive model. A crucial part of this phase is to secure the implemented activities in order to make sure that this is not a one time only. Perform evaluation What was the result of the implemented improvement activities? In this phase attention is given to the extent in which the original objectives were met. On the basis of these observations, the way of continuation of the improvement process can be determined. Scope of high-level, low-level and integration testing In the TPI-automotive model a differentiation is made between three test levels: Low-level testing Integration testing High-level testing High-level testing is the process of testing whole, complete products. A whole, complete product means a product as defined in the assignment for this product. The purpose of high-level testing is to show to which extend the requirements are implemented in the system. Low-level testing is the process of testing individual components, one at a time or in combination. To properly test a component, in general, the tester is forced to use stubs and drivers or to replace these with the real component(s). Integration testing is the process to expose faults in the interfaces and in the interaction between integrated components. In contrast with low-level testing the main focus of integration testing is the functioning of the assembly. The scope of integration testing is changing during the complete development process. In the first stages of the development process the focus is on integration testing of components at the lowest level. The testing is not very much different from low-level testing. At the end of the development process it is integration testing of the assembly of complete subsystems where the character of integration testing is more like a highlevel test. Still for these two opposites the same key area (21) is applicable, because the processes of integration testing are the same. Although in the model a differentiation is made between the three test levels, this doesn t mean that for every analysis all three should be in scope. The usage of the model for a situation where the high level is in scope is straight forward. The key areas 1 to 19 are applicable for this situation. If also the low-level testing and integration testing are within the scope of the analysis key areas 20 and 21 are also applicable. In situations where two or more test levels spanning more than one organisation or organisation part, the usage of the model will be a little more complicated. There are Sogeti Deutschland GmbH. Version

9 two possible ways to use the model. In the first approach the test levels are analysed as a complete process and the results are aggregated into one TPI-matrix. In the second approach the test levels are analysed individually and the results of the analyses are presented in a separate TPI-matrix for each test level. For high-level and low-level testing this means that the key areas 1 to 19 are applicable and for integration testing this must be extended with key area 21. However, in practice the low-level testing will most of the time be taken into account when one or more of the other test levels are analysed. Sogeti Deutschland GmbH. Version

10 1 TEST STRATEGY One of the most important key areas is the test strategy. The aim of the test strategy is: Detecting the most important defects as early and as cheap as possible "Most important" is related to the risks for the organization if the product is of insufficient quality. The test strategy defines which requirements and (product) risks are covered by which test level. The better each test level defines its own strategy and the more the different test level strategies are adjusted to each other, the higher the quality of the overall test strategy. Test process improvement usually starts with the high-level tests. As a consequence, for this key area a test process passes through the different levels in maturity: starting with a risk-based test strategy for a single high-level test to a situation where an overall risk-based test strategy is established covering all test levels and evaluation. A feature of the start level is that the test is only controlled by resources and time. Usually only one test design technique is used and only functionality is tested. Also there is no coordination between the different test levels regarding the quality characteristics to be checked, the scope of the test, etc. The test strategy is based on risk estimation (risk based test strategy). This risk estimation offers the possibility to find the optimal balance between the desired quality and the amount of time/money required. Parts with a high risk will be in full focus of the test. The risk estimation in combination with the size of the part determines the amount of time/money required to test this part. Essential steps towards a risk based test strategy are: determine quality characteristics; In consultation with the stakeholders, the quality characteristics are determined on which the test is to be focused. During the test process, the test results of the selected quality characteristics are reported to the commissioner. determine the relative importance of quality characteristics; Based on the results of the preceding step it is indicated how to distribute the test effort among the selected quality characteristics. The starting point is that the testing of each quality characteristic is equally time-consuming. break down into system parts; In this step the system is broken down into system parts based on the logical system architecture. When possible divide the system such that each system part is related to one supplier. If deviations from this division are made, they must be explicitly motivated. Besides this the item "total system" is often taken into consideration to indicate the relative importance of testing the entire system. The aim is to cope with risks, which are not covered by testing all system parts individually. determine the relative importance of system parts; Based on the results of the preceding step, it is indicated how to distribute the test effort among the system parts. The starting point is that testing each system part is equally time-consuming. Next, indicate for each system part which quality characteristics are applicable and how thorough these should be tested, in relation to the assigned importance. If the decomposition has led to a one-on-one relation between system parts and suppliers, concrete entry and/or acceptance criteria can be communicated to the supplier. determine measuring techniques to be used; As a final step within the test strategy, measuring and in particular test design techniques are selected which apply to the selected quality characteristics and system parts. Sogeti Deutschland GmbH. Version

11 In defining the test strategy, specific points of attention in determining the risks are: The functionality of system parts, The complexity of system parts and their interface, The supplied proof of quality of system parts (via test cases, test reports, certificates, established user base, etc.) by suppliers or the desired proof of quality described in the entry criteria. The transparency of the system parts. Transparency means how easy it is to determine how the system part functions and is therefore an indication how easy it is to detect the cause of failures. Transparency can be established for instance by original specifications provided by suppliers of the system parts, by providing source code, by so-called test windows in the software to display intermediate results. The frequency of use of the system parts. The potential damage if a specific part of the software fails. The availability of (test) resources (both staff and (test) infrastructure) Starting point for the risk assessment part of the test strategy are the functional and non-functional requirements, possible standards or regulations and other constraints. Examples may be: performance requirements, legal requirements, safety requirements (e.g. ISO 61508), and memory constraints (ISO: Resource Utilization). Safety related systems or system parts are quite special in the discussion about risks. Although it is possible to make a differentiation between a high or low influence on safety not testing such parts is out of the question. The thoroughness of testing and used test design techniques are the instruments that can be influenced by the risk estimation (although ISO will limit the possibilities). 1.A Test Strategy for single high-level test The commissioner of a test expects certain qualities of the system to be delivered which are very different for each system. It is of great importance that the commissioner and the supplier can communicate with each other about this, and, depending on the commissioners demands, make a translation to a test approach. Discussing the test strategy with the commissioner is often regarded as a revelation. Suddenly, all sorts of choices are available, instead of just for time and money. A well established test strategy contributes to a more manageable test process. The information that is generated during a risk based test strategy is of valuable use to make a well considered choice between demanded quality, time and money. When a commissioner wants testing to be cheaper, the test manager can provide the commissioner a choice: should a particular test be omitted or should another test be performed lighter? This gives the commissioner the opportunity to see the consequences of less time and/or money available. Example I: A car manufacturer will implement a new version of existing software. For this new version an acceptance test is planned and therefore a risk based strategy will be carried out. In the preparation of this strategy it becomes clear that the new version is not really a new piece of software but a newly parameterized version of the old software. This information leads to the conclusion that risks introduced by code changes don t exist and therefore no low level testing is necessary. In this case the risk estimation can focus on which requirements are related to the parameter settings changes and for which requirements a regression test is necessary. This analysis saves a lot of time and effort and made it possible to focus on the risks that are really necessary to be covered. Sogeti Deutschland GmbH. Version

12 Checkpoints 1.A.1 1.A.2 1.A.3 1.A.4 1.A.5 1.A.6 A motivated consideration of the product risks takes place. Typical risk categories to be verified are: technical risks (a FMEA can be used as basis), organizational risks associated to development/test, operational usage of the product, political and contractual risks and liability. The consideration implies at least the following aspects: Regression testing of unmodified parts of the software is part of this strategy when the test object is an update or new release of existing software. Software is often parameterized, for instance because of different country legislation. In such cases, part of the test strategy should be whether and how the different parameter settings are tested, based on the estimated risks. The software to test can be commercial-off-the-shelf (COTS), reuse or core, tailor made and often a combination of these. The test strategy must take into account the different risk profiles of COTS, reuse/core or tailor made software. Risks starting from the dependencies of products of the HW- and SWbaselines are taken into account, e.g. required backward compatibility or HW breaks. The stakeholders of the product are involved in the process of defining the test strategy. At least the stakeholders (especially the acceptant of the product) have to be invited to review the proposed test strategy and its present status. There is a differentiation in test depth, depending on the risks and, if present, on the acceptance and entry and exit criteria: not all system parts, variants and versions are tested equally thoroughly and not all quality characteristics are tested (equally thoroughly). When incremental delivery of functionality takes place (so-called A-Sample, B-Sample, etc) for every increment a clear decision is made what must be tested and which regression tests must be carried out. One or more test design techniques are used, suited for the required depth of a test. For retests also a (simple) strategy determination takes place, in which a motivated choice between "test solutions only" and "full retest" is made. At least a differentiation between changes in parameter settings and changes in source code is made. Dependencies Test design techniques, level A, Informal techniques Test design techniques are necessary to make the choices for lighter or more thorough testing concrete. Commitment and motivation, level A, Assignment of budget and time Strategy needs to be discussed with the commissioner of the test, because it is strongly related to the time and money needed. Improvement suggestions Involve the various interested parties such as commissioner, product line manager, and project manager in determining the test strategy. Create awareness by indicating the risks of the current working method, or indicate how testing can be carried out cheaper and/or faster. Sogeti Deutschland GmbH. Version

13 If there is only one design technique in use, then try to fit in more and/or less depth by means of simple variations. An example of more or less depth is testing or not testing boundary values. Boundary values are values which closely surround the limit of a range. In the condition "A>=10", 9, 10 and 11 are boundary values. For retesting, formulate a working method in which each time the consideration between a full retest, a "thin" retest (per defect/function/subsystem) or even no retest is made. The result of this consideration shall be motivated and documented. For safety related software a full retest is most of the time mandatory. Distinguish system parts and quality characteristics and try to assign a relative importance to each system part and each quality characteristic. Translate this importance into a lighter or a more thorough test. In order to minimize the risk of losing time by a late delivery of certain system parts fall back scenarios can be considered. Replacement of a system part by a previous version with more or less the same functionality can help. The risk of regression (an already tested system part does not function properly anymore after the introduction of a new version of any other system part) should be considered in this case. In case of incremental development the determination and maintenance of a regression test will save a lot of time. The regression test can be build up gradually per increment. The usage of test tools should be considered in this case. Eventually perform the complete determination of strategy. For testing parameterized software, there are some possibilities: test the software using different parameter settings (does the software deal correctly with the set parameters) review or test whether the different parameter settings that are used, are correct (did we set the parameters right for each situation). When the amount of possible parameter settings is too high (because large variant ranges or a lot of regulation differences between different countries) a risk estimation is necessary to find out which parameter settings can be (partly) neglected during test. COTS has compared to tailor made software a different risk profile. As a general rule of thumb, for COTS the functionality risks tend to be smaller, as other users of the software probably have found many of the existing functionality defects already. On the other hand, the risk whether the software is suitable enough for the organization is higher, as it has been designed as a "one size fits all". Identify the commissioner of the test and let the commissioner make clear what is expected of the test. 1.B Combined testing strategy for high-level tests Coordinating the test strategies of the different high-level tests prevents that tests are performed twice or that necessary tests are disregarded. It offers the possibility to determine when the right moment is for performing a certain test. The right moment is the test level where the overall costs for testing plus the costs of rework and retesting are at the minimum. Precondition for this determination is that for each high-level test is known which tests are planned and how thorough testing is planned to take place. The quality of the communication between commissioner and suppliers is of the utmost importance here. Both of them must understand their mutual interest in demanding and providing information concerning tests conducted by the supplier. The commissioner has to ask for the right information and the supplier should give a clear insight in what he has planned to test and what not. Sogeti Deutschland GmbH. Version

14 Checkpoints 1.B.1 1.B.2 1.B.3 1.B.4 1.B.5 Coordination takes place between the different high-level tests, often the system-, acceptance- and production acceptance test, or supplier and commissioner side testing, in the field of test strategy (risks, quality characteristics, area of consideration of the test and planning). The result of the coordination is a coordinated strategy, which is documented. During the total test process this strategy is controlled. Each high-level test determines its own test strategy, based on the coordinating strategy, as is described in level A. Deviations from the coordinating strategy are reported. A substantiated adjustment of the coordinating strategy is made based on the risks identified for these deviations. The validity of the strategy is checked in case of incremental delivery for each increment. For retests coordination takes place between the different test levels. In case the different test levels exceed the borders of the organization, commissioner and supplier make clear decisions about the retest at both sides based on the entry and exit criteria. Dependencies Life cycle model, level A, Planning, Design, Execution For coordinating it must be agreed beforehand about what, how and when to test (phase planning). All activities to be carried out must be documented. This requires insight in the process, for which a life cycle model is a prerequisite. Test design techniques, level B, Formal techniques The coordination of the various tests demands more insight in the quality and the depth of each test. This means the use of more formal test design techniques. Commitment and motivation, level B, Testing integrated in project organization To establish the coordination, commitment from the project management is required. This management must therefore have a desire for insight into the quality and depth of testing. Communication, level B, Project communication (defects, change control) Coordination requires that there is communication throughout the test process between the test levels and with the rest of the project. Test process management, level B, Planning, execution, monitoring and adjusting Insight into the quality of each test means that relying on plans only is not sufficient; the test process should be monitored and adjusted. (if applicable) Integration testing, level B, Test strategy for integration In case integration testing is high-level oriented: this is one of the high-level tests that shall be part of the overall test strategy. Improvement suggestions Begin with obtaining insight in what the different tests do. Important here is the extent to which insight can be obtained into depth and completeness of the individual tests. Indicate possible risks. Often improvement of the test process starts with a certain test level and there is no coordination yet with other test levels. In determining the strategy of this test level, define what is expected of other tests (in the area of test coverage). Try to find obvious deficits or double testing and initiate a discussion about them. Sogeti Deutschland GmbH. Version

15 Implement the role of test coordinator whose task it is to coordinate test levels, to record them in a master test plan and to monitor the coordination. The test coordinator reports to the project manager and the commissioner of the test. To prevent an entanglement of interests, it is preferred that the test coordinator is independent in relation to the various test levels. It can be considered to establish an acquisition inspection, in which testware of a certain test level is evaluated by another party on completeness and correctness. For example the software of an ECU is delivered to the OEM with test plan, test cases including trace-files (if relevant) and test reports. The OEM evaluates the testware to ensure that the entry criteria are fulfilled. Additional to the check on testware a so-called entry test can be defined and communicated to the deliverer of the test object. As part of the acceptance of the delivered test object this test is executed. This test shows clearly and objective whether the test object is ready for the next test level. Make use of exit and entry criteria per test level, where explicitly is asked for documentation and traceability of test coverage. Define entry and exit criteria for each test level to make a clear separation between the different test levels. Typical entry criteria are no defects left with the highest severity category, or prototype car is available in the defined configuration. Typical exit criteria are all planned test cases are executed and no pending defects left anymore. In the case of much overlap between certain test levels it can be an option to combine both test levels, i.e. combining the system test and acceptance test to an integrated test. Points of interest are the responsibilities and expectations. Legitimate reasons for the organization of the integrated test are: the suitability or availability of the test environment; early diagnosis and especially the correction of relatively important defects in the development process by using acceptance test cases; early exchange of knowledge about the system by developers and the commissioner; the common use of the test environment and the related management procedures; the use of test tools which normally cannot be used in the as-if-production environment, are at the disposal of the commissioner, including technical support; timely transferring knowledge to the commissioner about testing; early involvement in testing will stimulate the commissioner to actively think about testing; the integrated test optimizes the use of resources; personnel and test facilities are deployed from one source and conflicts of priorities are prevented in time; placing under one management improves the mutual understanding and the communication between the parties. Example: For new functionality an OEM has given the assignment to develop a new ECU for a brake system. The ABS of the brake system will be enhanced by an electronic brake system which can replace the hand brake. The ABS and the ECU hardware will be developed by the supplier who developed this already for a previous model. The electronic brake system software is developed by another supplier. This supplier can t carry out his system test on the target hardware because the delivery is planned much later in the project. The OEM and the software-supplier decide to combine the system test and the acceptance test. The software-supplier will design test cases for the system test and these will be reviewed by the OEM. The system test will be carried out by OEM and software-supplier together. The OEM will combine these tests with its own test cases for the acceptance test. By combining these tests the test environment (prototype car) can be used very efficiently. The combined execution of the test cases has the advantage that the understanding of the system by the OEM is enhanced and that defect analysis becomes easier by the direct support of the software-supplier. Although the test has to start with a software product that hasn t the right quality to Sogeti Deutschland GmbH. Version

16 start the acceptance test, the reality showed that the combined system and acceptance test led to a product that was accepted with a minimal risk by the OEM in less time than necessary if both tests were carried out separately. 1.C Combined strategy for high-level tests plus low-level tests or evaluation By involving the low-level tests or the evaluation levels in the coordination, yet more possibilities for optimization are available. The advantages of low-level testing are: they require little communication due to the fact that often the one who detects the defect is both the one who created the defect as well as the one who solves the defect. the tests detect defects in an earlier stage of the system development cycle than the high-level tests. Evaluation compared to testing means detecting more in less time and earlier in the development process. However, not everything can be evaluated, that is why testing remains very important. Expanding the scope of coordination from testing to evaluation provides many extra possibilities for optimizing the effort. Especially nonfunctional quality characteristics such as maintainability, portability, or reliability can often better be evaluated than tested. Involving either evaluation or low-level tests in the total test strategy is profitable to the total test process, because this creates more possibilities to optimize the strategy, i.e. detecting the most important defects as early and as cheap as possible. Checkpoints 1.C.1 1.C.2 1.C.3 1.C.4 1.C.5 1.C.6 Coordination takes place between the high-level tests and the low-level tests or the evaluation levels in the area of test strategy (risks, quality characteristics, area of consideration of the test/evaluation and planning) The result of the coordination is a coordinated strategy, which is documented. During the total (evaluation and) test process this strategy is controlled. Each high-level test determines, on the basis of the coordination, its own test strategy, as described in level A. (if applicable) Each low-level test determines, on the basis of the coordination, its own test strategy, as is described in key area "Low-level testing", level C. (if applicable) Each evaluation level determines, on the basis of the coordination, its own evaluation strategy, as is described in key area "Evaluation", level C. Deviations from the coordinating strategy are reported. A substantiated adjustment of the coordinating strategy is made based on the risks identified for these deviations. Dependencies (if applicable) Low-level testing, level C, Low-level test strategy Low-level tests have to be executed following their own test strategy, which is determined by the coordinating test strategy. Sogeti Deutschland GmbH. Version

17 (if applicable) Evaluation, level C, Evaluation strategy Evaluations have to be executed following their own evaluation strategy, which is determined by the coordinating test strategy. (if applicable) Moment of involvement, level C, Start of requirements definition (if applicable) Integration testing, level B, Test strategy for integration In case integration testing is low-level oriented: this is one of the low-level tests that can be part of the overall test strategy. The total strategy must be determined at an early stage. Improvement suggestions Start communicating the coordinating strategy of the high-level tests with the party responsible for the low-level tests (=developers, most of the time of the supplier) or the evaluations (= often quality assurance of the supplier). This communication will almost always exceed the boundaries of the organization. Find overlap and holes in total coverage between all these tests and evaluations. The commissioner has to decide together with the supplier of the test object which test should be carried out when. The quality of the system (part) to be delivered can be raised if early in the project a prototype car is available to use rapid-prototyping techniques by the supplier. Tests defined in this stage of the project can later be used in the system or acceptance test with the real software and hardware. Find out if it is better that certain high-level tests are performed during low-level tests or evaluation and vice versa. The verification of norms and standards of the software (quality characteristic: maintainability) can for example be part of a test or an evaluation. Example: During the development of an Adaptive-Cruise-Control-System (ACC-system) the tests are mutually coordinated under the responsibility of a test coordinator. The system consists of an ACC-ECU, engine-ecu, ESP-ECU and transmission-ecu, which are developed by four different suppliers. The OEM will integrate the four ECU s and test the integrated system. To be sure that he gets the right quality a risk-based test strategy is carried out with the suppliers. Decided is what the focus should be of the tests carried out by the suppliers. Based on the risk-based test strategy entry and exit criteria are formulated. As part of the entry criteria an entry test is formulated for every ECU. This test is carried out by the OEM and the description of this test is communicated to the suppliers. If the ECU doesn t pass the test it will be sent back to the supplier. The requirements will be described together with the suppliers. The suppliers will break down these requirements into detailed software and hardware specifications. The specifications will be reviewed based on what is described in the risk-based test strategy. In this formal review round the OEM is directly involved. With this approach the OEM minimizes the risk that ECU s are not fit to integrate into an ACC-system and that the integrated system will still contain severe defects. Low-level tests as well as evaluation levels start earlier than high-level tests. In the coordination this has to be taken into account. Combine the inspection of the test basis (see also key area Life cycle model, level B) with evaluation of the specifications. Exchange testware between the different test levels, for example between the acceptance test and unit test. The advantage is that certain tests do not have to be prepared twice. Beware that the tendency can arise not to prepare or execute one's own test so that the testware takes over the role of the specifications ("if the test cases are processed well, the system is built well"). Sogeti Deutschland GmbH. Version

18 1.D Combined strategy for all test and evaluation levels This description is similar to the preceding level C, but now there is coordination between high-level tests, low-level tests and the evaluation levels, to achieve further optimization of the total test and evaluation strategy. Checkpoints 1.D.1 1.D.2 1.D.3 1.D.4 1.D.5 1.D.6 Coordination takes place between the high-level tests, the low-level tests and the evaluation levels in the area of test strategy (risks, quality characteristics, area of consideration of the test/evaluation and planning). The result of the coordination is a coordinating strategy, which is documented. During the total evaluation and test process this strategy is controlled. Each high-level test determines its own test strategy on the basis of the coordination, such as described in level A. Each low-level test determines its own test strategy on the basis of the coordination, such as is described in the key area "Low-level testing", level C. Each evaluation level determines its own evaluation strategy on the basis of the coordination, such as is described in the key area "Evaluation", level C. Deviations from the coordinating strategy are reported. A substantiated adjustment of the coordinating strategy is made based on the risks identified for these deviations. Dependencies Low-level testing, level C, Low-level test strategy Low-level tests have to be executed following their own test strategy, which is determined by the coordinating test strategy. Evaluation, level C, Evaluation strategy Evaluations have to be executed following their own evaluation strategy, which is determined by the coordinating test strategy. Moment of involvement, level C, Start of requirements definition The total strategy should be determined at an early stage. Integration testing, level B, Test strategy for integration. Improvement suggestions Appoint a test/evaluation coordinator (TEC) who coordinates the evaluations and tests and who also continuously monitors this coordination. The TEC reports to the project manager and possibly also to the commissioner. Point of interest is the independence of the TEC. Make the test and evaluation plan an integral part of the (system development-) project plan. If suppliers are working with sub-suppliers test levels and evaluations of the subsuppliers should be considered in the overall test strategy. In this case the same procedures that are implemented between commissioner and supplier can be used between supplier and sub-supplier. The supplier is responsible that the commissioner gets the right information to build an overall test strategy for the complete development process of his desired product. Sogeti Deutschland GmbH. Version

19 Take into consideration the total costs and benefits of testing from development till deployment to make a well thought decision about what risk to test when with an optimal balance between benefits and costs. Sogeti Deutschland GmbH. Version

20 2 LIFE CYCLE MODEL Within the test process a number of phases can be distinguished, such as planning, preparation, design, execution and completion. Each phase consists of a number of activities, of which the aim, the products to be delivered and way to implement the activity are described. A life cycle model makes the test process manageable. It becomes clear who has to do what and when and the different activities can be planned and monitored in cooperation of the parties involved. When this transparency is lacking, activities are executed either too late or are forgotten, activities are running out of time because no insight in activity duration is available. There is no insight in the progress of the testing process and because of this finishing the process in time is almost impossible. Finally, the test process probably stays longer than necessary on the critical path of system development. 2.A Planning, Design, Execution The most important phases in the test process are Planning, Design and Execution. The main activity of the Planning phase is defining a test plan. In a test plan it is defined how, by whom, with what, and when the test activities are to be executed. The main activities of the Design phase are defining the test cases and preparing the test execution. In this context, also the necessary test infrastructure is realized. The aim of the Execution phase is to perform the specified tests to gain insight in the quality of the test object. Checkpoints 2.A.1 2.A.2 For the tests (at least) the following phases are recognized: planning, design and execution. These are subsequently performed, possibly per subsystem. A certain overlap between the phases is allowed. Activities to be performed per phase are mentioned below. Each activity contains sub-activities and/or aspects. These sub-activities and/or aspects are meant as additional information and are not obligatory. For the Planning phase: Activity Sub activities / aspects Product - formulate assignment - determine the test basis - commissioner and supplier - scope - aim - precondition - starting points - Determine relevant documentation; like system requirements, software requirements, software design and other documents that are used to derive test cases - identify documentation - defined in test plan - defined in test plan Sogeti Deutschland GmbH. Version

21 - define scope (area(s) of consideration): what will and what will not be tested - - defined in test plan - determine test strategy for this iteration - strategy determination - estimating - defined in test plan - set up organization - identify test deliverables - define infrastructure and tools - define test management - determine planning (aspects like activities, dependencies, milestones, start & end dates and needed resources) - determine required functions - allocate tasks, authorizations and responsibilities - describe organization - allocate personnel - determine training - determine communication structures - determine reporting lines - determine test products - set up norms and standards - define test environment - define test tools - define office environment - define infrastructure planning - define test process management (progress, quality, reporting) - define infrastructure management - define test product management - define defects procedure - defined in test plan - defined in test plan - defined in test plan - defined in test plan - define general planning - defined in test plan Sogeti Deutschland GmbH. Version