Journal of Engineering Technology

Transcription

1 Applied practices of Test Maturity Model (TMM) for small and midsize test organizations Kidu Kim 1, Byungkook Jeon 2* and R. Young Chul Kim 3 1 Telecommunications Technology Association,Nuritkum Square, 396, World Cup buk-ro, Mapo-gu, Seoul 2 Dept. of Software, Gangneung-Wonju National University, Wonju City, Gangwon Prov., 26403, Korea 3 SE Lab., Dept. of Software and Communications Engineering, Hongik University, Sejong Campus, 30016, Korea Abstract: Software is being used in various industries like automobiles, medical care, aviation, and defense, which specially requires high quality [1]. Most of software development companies have been considering quality seriously. They are still looking for diverse ways to improve the quality of their software products such as removing errors with many tests, adopting particular software methodologies or processes, and verifying & validating them. In the case of Korean small and midsize enterprises (SMEs), it is difficult to use diverse methods because of their tight development schedule and insufficient human resources. For SMEs, CMMi and SPICE are also burdensome models for improving their qualities. To solve this problem, we propose to develop a simplified Test Maturity Model (stmm) [2, 3]. Through assessing with this stmm model, we suggest how to support test maturity of any test organization. Even though we have completely developed this model, we can t directly apply to domestic SMEs. This is some applied practices to check possibilities of applying the stmm to four domestic SMEs for improving their test maturities. Keywords: Test Maturity Model, Improvement Test Process, Software Quality, Capability Maturity Model 1 Introduction As the fourth industrial revolution has been beginning in Korea [4], annual usage of software is exponentially growing in various industrial fields. Since 2010, software usage in automotive industry has been increased by 50 percentages [1]. So the importance of software quality has still been increasing. Accidents such as train collision at the Sangwang-ri station in Korea and Toyota s pedalgate raise public awareness of the software quality. In order to reduce software problems, software engineers work hard to develop software, which is not easy to improve the quality of its software formless. Many researchers have been trying the following ways to improve software quality: 1) testing software perfectly to eliminate errors, 2) applying software methodology or formal process in development, and 3) assessing with proven certifications or models. But still perfect testing is practically impossible [5]. Constructing its software process is difficult in domestic SMs development situations with a short time and no enough budge cost. Major software development companies are easily applicable to assess with proven certifications or models, but SMEs cannot spend their costs to assess their capabilities. To solve these problems, we propose to simplify the 157

2 original TMM" for SMEs in Korea. We apply our simplified TMM model to four software development SMEs which are assessed with our suggested activities through the process of diagnosis, and then finally guide to improve test maturities of their test organizations In this paper, chapter 2 describes the related work of "Simplified TMM (stmm)". Chapter 3 describes the process and method of applying the stmm. Chapter 4 explains the results of assessing with the stmm, and the guide for improving the test maturity of the software development company. Finally, the conclusions and future research are mentioned. 2. Related Work Although there are many diagnostic models for quality improvement, we select a model that is not commercial and has customizable for SMEs in Korea. The first original model is the Test Maturity Model (TMM). The reason for choosing TMM is that we can directly customize the model to diagnose and improve the test quality capability of the test organization. The original TMM is software testing maturity model developed by Dr. Burnstein of Illinois Institute of Technology (IIT) in Based on the concept of CMM [6], this model provides test maturity levels for improving the test capability [7]. The second model is a recently proposed Test Maturity Model integration (TMMi) [8]. This model is more complicated to use in SMEs in Korea than the original TMM. Table 1 shows comparison of CMMI, TMMI & Simplified TMM. With the TMM and TPI models, we suggest developing our model for SMEs in Korea, which simplifies the focused on software testing origination [9]. The reason why we chose these models such as TMM and TPI is useful for SMEs which are preparing for specific process improvement and self-diagnosis. Table 1. Comparison of CMMI, TMMI & Simplified TMM [9] Model CMMI TMMi Simplified TMM Year of development In development Developer SEI TMMi Foundation TTA/Hongik University Model type System Engineering, Software development and maintenance process Software Test Software Test Maturity level of organizationlevel maturity & respective process area Organization-level maturity Organization-level maturity Process Area Method (developed up to level 2~3) Procedure SCAMPI-based Questionnaire, interview and official/unofficial Questionnaire, interview and onsite Characteristics SCAMPI-based of corporate competency of test maturity via official/unofficial of test maturity in compliance with procedure 158

3 Model CMMI TMMi Simplified TMM Certification target Software Process Software Test Process Software Test Process The TMM reflects the maturity growth through 5 levels well known from Software Engineering Institute s Capability Maturity Model. The internal structure of the TMM consists of test techniques to support a quality testing process in a systematic way, which consists of maturity goals of each level with identifying testing. Each goal is supported by sub-goals which are achieved by performing activities / tasks / responsibilities. Activities / tasks / responsibilities are categorized into their views of managers, developers / testers, and users / clients. The TMM levels have included from the undefined level 1 to the optimized level 5 for all activities The second model is the Test Process Improvement next (TPI next) [10], which is an improved model of the TPI model developed by SOGETI. We adapt the TMM (that mainly refers to the test activity) with the TPI next (that is used to supplement the test process). The structure of TPI next as follow Figure. 1 consists of key areas, maturity levels, checkpoints, and clusters. The key areas represent the relevant skills, tools and reports in the test process. The maturity levels are focused on testing each key area and classifying maturity levels. The test maturity matrix shows the level of the testing process, which is determined through all key areas. The checkpoint can objectively assess key areas. An improvement proposal is an improvement plan to improve the test process over the measured level. A cluster is a set of checkpoints in several key areas. Finally, the enablers synchronize the software lifecycle and other processes to maintain the improvement of the test process. Figure. 1. The Structure and Level of TPI next Figure. 2 shows the development process of "Simplified TMM (stmm)". In the model development stage, we analyze the structure of authentication models, and define the structure of the "Simplified TMM (stmm)" suitable for our domestic situation [11, 12]. We also analyze the activities / tasks / responsibilities of TMM to identify the key elements of the "Simplified TMM 159

4 (stmm)". Based on this process, we extract detailed activities related to key elements. The derived key elements are judged to be suitable for each in our domestic software development environment. Conformance can be "Conformance", "Non-conformance" and "Partial conformance", but we delete "Non-conformance in the "Simplified TMM (stmm). Figure. 2. Development Process of the Simplified TMM (stmm) In order to assess the level of test activities and procedures, we make the checklists based on the of the "Simplified TMM" in Figure. 3. In order to easily understand the test maturity level of the target organization, the level areas are divided into test process, test management and monitoring, test environment, test organization, and test technology to understand the good areas and lacked areas of the target company. In the level area, there are 1~4 detailed. The level area provides a basis for a clear of the insufficient level area of the enterprise. 1. The Simplified TMM (stmm) Level Maturity goal (Result of 2.1 ) Area Result of judgment by detailed Detailed 1 D 50 2 G 80 3 D 50 5 D 50 6 D 50 1 G 80 2 G 80 3 G 80 1 G 80 2 E 100 Result of judgment by PA FA FA

5 G FA 80 [Omitted] 3. Percentage score by level area Maturity level Test process Test management and monitoring Test environment Test organization Test technology Level % 100 % % 100 % % Level % % % % % Level % % % % % Level % % % 9.77 % 9.02 % Figure 3. Checklist of the Simplified TMM (stmm) [3] Table 2. Criteria for judging Items Criteria for judging Description FA (Fully Achieved) Detailed achieved the (86% ~ 100%) LA (Largely Achieved) Detailed achieved most of the (51% ~ 85%) PA (Partially Achieved) Detailed achieved some of the (16% ~ 50%) NA (Not Achieved) Detailed did not achieve the (0% ~ 15%) Table 3. Detailed Criteria for judging Items Criteria for judging Description E (Excellent) The test activity (product) performed the detailed. G (Good) Some of test activities (product) were not performed for the detailed D (Deficient) Most of test activities (product) were not performed for the detailed 161

6 N (Nothing) Activities were not performed In Table 2 and Table 3, there are the criteria for the of software development organizations. In the existing method, each activity can be Pass or Fail. To determine whether it is Pass or Fail, it may be necessary to be able to distinguish between Pass and Fail. Therefore, we have FA / LA / PA / NA to determine for each maturity sub-goal. However, it is difficult to distinguish Pass and Fail for each detailed because each company has different activities to perform. To solve this problem, we define Excellent, Good, Deficient, and Nothing as the criteria for judging for each detailed, and also score the Criteria of judging for each. This is used as a basis for determining. Based on the results of the detailed, the are finally diagnosed. 3. of the Simplified TMM based Test Maturity We describe the application of Simplify TMM (stmm) to domestic software development companies. This model includes the procedures and criteria for diagnosing the test level of the software development organization. Figure. 4. Procedure of Simplified TMM (stmm) [2] The procedure is composed of four steps as Figure. 4. In the Application for Diagnosis step (Step 1), the reviewing agency receives the report result of diagnosis submitted by the target company. The reviewing agency reviews the result. If appropriate, we determine whether or not to examine it. Then establishes a diagnostic plan. In the Diagnostic plan stage (Step 2), the visiting schedule is adjusted through consultation with the target company. Assessors are classified into lead assessors and general assessors. We have formed a diagnosis team with one lead assessor and two general assessors. The lead assessor becomes the leader of the diagnosis team. This team asks the evaluation materials necessary to target company for site review and pre-review through the preparatory meeting. Based on the evaluation materials, an on-site evaluation plan is established through a preliminary review meeting, and is notified to the target company. Documents required at this step include plans, pre-questionnaires, document review lists, and pre-review minutes 162

7 etc. In the Interview and Product Review step (Step 3), the diagnosis team visits the target company, and reviews the product and interviews employee. Based on this procedure, the level is diagnosed, and is prepared with the on-site evaluation results. The documents required at this step include the test maturity sheet, the meeting data, the document review list, the document management list, the necessary information recording sheet, the interview questionnaire, the data, and the result report draft etc. In the step of deriving the improvement (Step 4), the diagnostic agency derives improvement based on the evaluation result report, and assesses the final diagnosis result. The diagnostic agency provides the final result report to the target company, and all schedules are terminated. Table 4 below shows the detailed activities and outputs of the procedures. Table 4. Action and Product Step Activity Product 1. Application for diagnosis 2. Diagnostic planing 3. Itreview and Output Review 4. Deriving improvements Tasks Receive an application Establish a diagnostic plan Select a target project Request related organization chart Request personnel selection Consult on interview schedule Pre-survey Select evaluation level Consult on meeting schedule Form an evaluation team Review pre-survey Personnel interview (development team, quality team, etc.) Review related documents Write evaluation results Analyze interview and survey Analyze document review results Report diagnostic results Diagnostic application Diagnostic plan Evaluation materials Organization chart Person in charge Pre-survey Evaluation level Diagnostic plan Pre-survey Interview form Product by level Summary of evaluation result Deriving improvements result 4. Results of the Simplified TMM (stmm) application We apply the Simplified TMM (sttm) to the domestic software development SMEs by the procedures and methods mentioned in Chapter 3. Table 5 shows the results obtained by actual application. The assessed companies are small and midsize software development companies (stmm) with 20 to 25 employees in Korea. They are companies with development experience in a variety of areas such as package software, web services, and SI projects etc. We do use anonymized names of the companies participating in the diagnosis and application. 163

8 Table 5 Action and Product - Level 2 Result of judgment by detailed Result of judgment by Area Detailed D 50 2 G 80 3 D 50 5 D PA D 50 1 G G 80 3 G 80 1 G 80 2 E FA FA G FA 80 1 G D PA G 80 2 D 50 3 E LA E D 50 7 N NA D PA

9 Result of judgment by detailed Result of judgment by Area Detailed G 80 1 D 50 2 D 50 1 D 50 4 G 80 5 D 50 6 D 50 2 G 80 3 D 50 4 E D 50 7 N 0 8 N 0 9 N 0 10 E E PA NA PA NA PA NA

10 Result of judgment by detailed Result of judgment by Area Detailed 3 D 50 6 N 0 7 N 0 - Level Area Result of judgment by detailed Detailed 6 N 0 7 N 0 2 D 50 6 D 50 7 N 0 8 N 0 9 E N 0 11 G D 50 Result of judgment by NA NA

11 Area Result of judgment by detailed Detailed 13 D N 0 17 D 50 6 N 0 7 N 0 Result of judgment by NA NA NA NA NA NA

12 Area Result of judgment by detailed Detailed 5 D 50 6 N 0 1 D 50 2 G 80 3 D 50 4 D 50 3 D 50 4 D 50 6 N 0 7 D 50 8 N 0 9 D 50 6 N 0 7 N 0 Result of judgment by LA NA NA NA 0 As shown in level, Level 2 is not satisfied. Levels 4 and 5 do not apply because level 2 is not reached. Based on the contents of Table 5, we summarize the deficiencies of each test field to help understand the vulnerability of the company to be diagnosed. Table 6 shows the diagnose results for each diagnostic field [13]. 168

13 Table 6. Result of Test Fields Test process Test management and monitoring Test environment Test organization Test technology Diagnose results Start the test at the completion of product development Do not set goals for each test level when performing the project No life cycle defined for defect after test Some projects do not establish test policies when creating test plans There is no procedure for the defect or issue that occurred after the test is performed There is no verification procedure for test preparation before test for each level No adequate testing procedures for project success Do not check resources required at all levels for testing There is no job definition for deatiled activities of each test Testers participate in only some steps of the project No changes or revisions possible for goals and policies established for project implementation There is no management data such as test logs for test management Developer and tester do not review the content together when reviewing a test plan There is no test data (eg test summary report) for test management There is no management method for test environment (SW, HW) Cooperation among the staffs is inadequate to deal with the test-related problems No resource management (personnel, test bed, etc.) is made to perform the test Mentoring activities for testing are insufficient compared to development activities No training and tools are prepared for testing There is no storage for product management after testing No template and related tools are preapred for test planning No support is made for new test tools No test tools for monitoring and managing testing are prepared Testers do not participate in resource (cost) management required for testing No organizations for smooth testing activities are not prepared There are no testing templates available for testers Testers do not participate in test policy establishment Testers do not participate when reviewing the test plan The quality of the test result is not managed because there is no quality manager Some test-related products are not generated because there is no test organization Some step-by-step tests (eg integration test, etc.) are omitted because there is no test organization Not enough support for test organiz No tester involved in project policy decisions Reporting system and cycle by person in charge are not clear Insufficient employment of new members such as test experts, test managers, etc. There is no way to manage performance results by test step. Defect tracking metrics are underutilized There is no black box (or white box) test design method when testing When preparing a test policy, it is not established by test step Does not support support for test design (eg: tools) Inadequate basic testing techniques and tools applied when performing a project Test results and test design data are not thoroughly reviewed Traceability between requirement and test case is insufficient Based on Table 6, we explain the lack of activities to improve the quality. Finally, we summarize all the contents and provide the result type document. 169

14 5. Conclusion & Future work In this paper, we develop and apply the Simplified TMM (stmm) for SMEs. Through this process, we are able to diagnose the test level of the software development company to discover the deficiencies, and improve the test maturity level using the diagnostic results. As a result, the target company hires the staff for quality support and trains them to acquire skills for defect tracking for improving the test maturity of the target company. We are currently conducting additional diagnoses for two SMEs, and improve the Simplified TMM (stmm) with various cases in the future. Acknowledgments This research was supported by Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Education(NRF-2017R1D1A3B ), and supported by Development of Information Communication and Broadcast R&D(2018 Open OS Environment Development and Diffusion) through NIPA(National IT Industry Promotion Agency)(S ), and supported by a grant(18ctap-c ) from Technology Advancement Research Program funded by Ministry of Land, Infrastructure and Transport of Korean government. References [1] Il Jun Park, Software Innovation Strategy for Creation Economy, Policy and issues, KEIT, [2] Hongik University, TTA, "A Study on Development of Korean Test Maturity Model I", Research Project Report, [3] Hongik University, TTA, "A Study on Development of Korean Test Maturity Model II", Research Project Report, [4] Jin Ha Kim, Seeking strategic responses to future social change ", KISTPEP InI(15), [5] Cem Kaner, Ph.D., J.D. Impossibility of Complete Testing, Law of Software Quality Column, Software QA magazine, [6] Mark C. Paulk, Bill Curtis, Mary Beth Crisis, Charles V. Weber, "The Capability Maturity Model for Software", SEI (Software Engineering Institute (SEI), [7] Ilene Burnstein, "Practical Software Testing", Springer, [8] TMMi Foundation, "Test Maturity Model integration (TMMi), Release 1.0", [9] Bo Kyung Park, So Young Moon, Kidu Kim, Woo Sung Jang, R. Young Chul Kim, C. R. Carlson, Refining an Assessing Model for Simplified TMM, PlatCon, [10] Gerrit de Vries, "The What and How of Testing TPI next and TMap Next Related", Sogeti, [11] Kidu Kim, "A study on enhanced test maturity model(tmm) through test process improvement", Hongik University Master's Thesis,

15 [12] Kidu Kim, "Improving heterogeneous maturity models based on assessing good software(gs) certification model ", Hongik University PhD thesis, [13] Eun Young Byun, Hyun Seung Son, Byungkook Jeon, R. Young Chul Kim, Reusability Strategy Based on Dynamic Reusability Object Oriented Metrics, Journal of Engineering Technology, Vol.6, Issue 1, pp , January * Corresponding Author : Byungkook Jeon, R. Young Chul Kim 171