Very Small Enterprises (VSE) Quality Process Assessment

Transcription

1 Very Small Enterprises (VSE) Quality Process Assessment JEAN-MARC DESHARNAIS MOHAMMAD ZAROUR ALAIN APRIL École de Technologie Supérieure, Department of Software and IT Engineering 1100 Notre-Dame Ouest, Montréal, Québec H3C 1K3, Canada Abstract The software industry is an important economic activity in industrial countries. Nowadays the investment in this field measured in millions of dollars, the IT organizations varies in their size and role. An area of concern to IT organizations is implementing and adopting Software Process Improvement (SPI) models. For many years, researchers have developed several SPI model, most of which like CMM, CMMI 1 and SPICE (ISO/IEC 15504) have been developed to fulfill large organizations needs, but they do not take into consideration the Very Small Enterprises 2 (VSE) needs. This has encouraged a number of researchers to study the SPI for VSE and try to find suitable models that fit to VSE needs. Our document explores the main processes concerning the VSE to discover which processes are the weakest, which we found to be the quality assurance, and shed light on it. This was shown by three rounds of consecutive assessments with about 10 VSE assessments each time. It confirms previous studies that have already done [4] [10] [11] [14]. Keywords: Software Process Improvement, Maturity Model, CMM, CMMISM, SPICE, OWPL, Micro- Evaluation. 1. Introduction In the past, the organizational improvement efforts were focused on product rather than process. It has been recognized that many management and support activities are needed to produce effective software organizations, for example, cost and schedule problem as 1 CMM and Capability Maturity Model have been registered with the U.S. Patent and Trademark Office. CMM Integration and CMMI are service marks of Carnegie Mellon University. 2 In this paper the words small organization, small enterprise, small companies are used interchangeably. well as, inadequate product quality are key results of inadequate management. These problems are well understood today by many organizations, since project managers neither have time nor resources to address management issues. It is also recognized that malfunctioning software development process will affects the quality of the resulted software product. As a result, the companies now pay great attention to improve their processes in order to improve their product quality. Since 1980s a large number of software process improvement (SPI) projects has launched world wide to satisfy companies needs to improve their processes. A number of models have been created and used as a result to these projects; the most popular are CMM [1] created by Software Engineering Institute at Carnegie Mellon University and ISO/IEC standard [3]. Since the structure and capabilities of small organizations differ than that for large ones, both CMM and ISO/IEC (and similar models) can t be applied in their current form in such organizations, since small organizations neither have time nor resources to apply such models, as well as, the long term Return of Investment (ROI) is not acceptable for such companies. Based on the differences in structure, motivations and goals between large and small organizations, the work in the SPI field can be divided into groups; the first group concerned in creating and improving software process assessment models that are used mainly by large and medium sized organization, these models are complex and consume a lot of time and cost. ROI for applying these models can not be grasped in a short term period. The second group concerned in creating and improving the software process assessment models for small organizations which must be simple, fast and less expensive than other models. 2. What does it mean Very Small Enterprises (VSE) The term VSE is used to describe very small organization in the field of IT. Unfortunately, there is no 1

2 standard definition for VSE size. The size is something relative, i.e. the organization having 100 employees considers another organization having 1000 employee as very large, while the organization having 1000 employee considers the organization having 100 employees as small one! And perhaps will consider the organizations having 10 employees as micro-organizations. Brodman and Johnson have made another definition for a small organization as the one that have fewer than 50 software developers [8]. Orci and Laryd [9] have developed a dynamic model of CMM to be applied in small organizations, in which they made different classification for organizations based on size, and introduced new terms such as: extra small and extra extra small, the following table summarizes their classification: Variant of Small XXS (extra, extra Small) 1-2 XS (extra small) 3-16 Number of People S (Small) Table 2 - Company sizes based on Orci and Laryd s classification [9] The Another definition for VSE introduced by Claude Laporte, Alain April and Alain Renault [12] says that the VSE is any IT services, organizations and projects with between 1 and 25 employees. It seems that the project should be excluded from this definition for two reasons: Firstly, none of the assessment work done for VSE is made at project level; assessments are used to be made at organization level. Secondly, the word project is not welldefined in the industry; as Paulk wrote [5] it could be an hour project with one employee or as defined by the PMI [15] a 3 years project with 40 employees. We will exclude the term projects from this definition and use it as the basis for our work from now on. 3. Methodology The group of researchers concerned in creating and improving the software process improvement models for small organizations focus their work in tailoring the CMM model, as well as, SPICE model to find the suitable process areas and set of practices that fit the needs of small organizations. One of these models is the OWPL (Observatoire Wallon des Pratiques Logicielles (Walloon Observatory for Software Practices) Gradual Framework originated at the University of Namur Belgium and applied in the Wallonia region in the same country. This framework aims at making small organizations aware of the quality issues and providing a continuous mechanism that supports SPI which will produce rapid but tangible results with minimum recourses [10]. The OWPL gradual Framework consists of three phases applied consequently as follows [11]: 1. Micro-Assessment The micro-assessment is the first step in the framework proposes to assess small organization. It is conducted through a simplified questionnaire called the Micro- Evaluation which first assesses the current software practices in the small setting, and then makes the organization aware of the quality aspects of software. This questionnaire covers six key practice areas which have been selected as the most pertinent to the targeted organizations (based on previous experience with small-setting assessments). These are: Quality assurance Customer management Supplier management Project management Product management Training and human resources management The questionnaire contains 18 openended questions, one of them provides a brief description of the organization being assessed, and another pertains the interviewer s remarks concerning the questionnaire itself. The remaining 16 questions cover all 6 main axis mentioned earlier, the responses being evaluated according to systematic analysis grids. The organization s practices are rated according to the assessed quality level of those practices and to the degree to which they have been implemented in the organization being assessed 2. OWPL Assessment Following the assessment, the collected data is analysed and results are set out in a report, which includes a brief presentation of the approach and a description of the organization being assessed, the data collected, data analyzed and results summary according to the six axes, a list of the main strengths and weaknesses according to SPI principles, and finally the practices recommended to help the assessed organization improve its existing practices and processes. The conclusions drawn from the Micro-Evaluation responses lead to the production of an action plan. A second Micro- Evaluation can be performed a few months after the first one, to investigate the progress achieved. 2

3 3. CMM/SPICE Assessment This model could be considered as tailoring model to the SPICE model with influence from CMM; it forms the main task of OWPL which aims at adapting the quality models, such as, CMM and SPICE to the context of small enterprises. Usually this evaluation model is sufficient for most of the small enterprises. Automn Adoption of Micro-Evaluation Although the Micro-Evaluation framework is developed for evaluating IT organizations in Wallonia area in Belgium, many other similar regions which have a lot of similarities with Wallonia area has applied this framework and got good results. As a result of the contract between Namur university in Belgium and ETS in Québec several evaluations have been accomplished in Québec area in 2004 and 2005 by undergraduate students as well as a graduate student [13] [14] under the supervision of one of the co-authors of this paper. A new research project has been founded in ETS called APPEQ (Amélioration de la Performance des Petites Enterprises Québécoises), that aims to help SMEs in the IT field improve their quality, productivity and performances. The authors have practiced the Micro-evaluation approach in ETS (Ecole de Technologie Supérieur) in Quebec Canada for several semesters, and build their work in this paper on three successive semesters; autumn 2005, winter and summer In autumn 2005 a set of 7 VSE has been assessed, in winter 2006 another round of assessment included 12 VSE; in the following semester (summer 2006) another round of assessment is made for 13 VSE. The results are recorded and analyzed for the total of 32 organizations. Our study does not aim at measuring the current improvement after the second or third round in the assessed organizations; our main goal is to explore the weakness points in the assessed organization to determine which topics are not applied well (Score < 2) in the VSE and shed light on them in our future work. For autumn 2005, the data shows that no topic can be considered to have weakness since all topics scored more than, the least scored topics (Score = ), as shown in figure-1, were the origin of quality and the product structure. Figure-1: Detailed Autumn 2005 Graph For the winter 2006, as in figure-2, several weakness points can be observed in topics: commitment to quality (Score=1.6), origin of quality (Score=), subcontractor selection (Score=1.8), verification (Score=1.9) and human recourse management (Score=1.4). Winter 2006 Figure-2: Detailed Winter 2006 Graph For the summer 2006, as in figure-3, several weakness points can be observed in topics: commitment to quality (Score=1.9), origin of quality (Score=1.8), subcontractor selection (Score=1.6) and subcontractors tracking (Score=1.8). Summer 2006 Figure-3: Detailed Summer 2006 Graph To have an overall view for the whole companies for the three semesters, we measured the average for each 3

4 of the 16 topic separately. The result, as in figure-4, shows that the main weakness points are concentrated in the commitment to quality (Score=1.9), origin of quality (Score=1.7) and Subcontractor selection (Score=1.8). The weakness in the subcontractor selection can be referred to the fact that this topic is not applicable for many VSE, since not all VSE are dealing with subcontractors. Average of all companies ( 32 Company ) 2. Code testing is performed mainly by the programmers in an ad hoc manner, i.e. no clear testing plans are used. 3. There are no specialized or trained employees that can apply quality management activities. 4. VSE depend on the personal skills for their employees in performing their tasks. 5. Most of the VSEs are not aware of the quality management activities. 5. Conclusion: Figure-4: Detailed Average graph (32 companies) To focus our discussion more deeply, we show in figure-5 below the average for the main 6 axis of the Micro-Evaluation. From this graph we can see that the main axis that is suffering from weaknesses is the quality management axis which scores 1.8 only. This paper has studied the data resulted from the application of the Micro-Evaluation assessment tool for three consecutive semesters in Quebec area, mainly to discover which axis is the weakest and discuss what reasons that let to this weakness. The obtained results lead us to the fact that the main axis that is suffering from obstacles and weaknesses is the quality management axis. This result opens the door for a set of research points that needs further considerations. 6. Future work: Our future work will focus in the following research points in order to provide a mechanism to improve the quality management axis of the software development process, and assess the results of the application of these mechanisms in VSE. Quality management Custormers Subcontractors Project management Product management Education and H.R. This result opens the door for other research points that needs further study and explanations, these points include the following: (A) management (B) management (C) (D) (E) management (F) Figure-5: Average of Scores for the main 6 axis Based on these results, it is obvious that the quality management axis is the weakest link that needs improvement in VSE, since it is the lowest scored axis. But what makes the quality management have such low score? Based on our study and analysis of the questionnaires results for the 32 companies, we can refer this weakness to several reasons as follows 1. Most of the quality management activities have been reduced to testing the code only. 1. We need to evaluate the product quality produced by VSE and check to what extent do these products affected by the quality management axis of the software development process? 2. When studying the product quality in VSE, how can we decide that the product quality is low? What measures we can use? 3. What kind of processes we can suggest to improve product quality, and what kind of tools we can use and implement in the VSE for this purpose? 4

5 4. After the application of the suggested tools in VSE, we have to assess the new result by revisiting the enterprises to assure that the quality is improved. 7. REFERENCES [1] M.C. Paulk, B. Curtis, M.B. Chrissis, and C.V. Weber, Capability Maturity Model for Software, Version 1.1 Software Engineering Institute, CMU/SEI-93-TR-24, February [2] SEI, Capability Maturity Model Integration (CMMI SM ), Version 1.1, CMMI SM for Software Engineering (CMMI-SW, V1.1) Continuous Representation, CMU/SEI-2002-TR-028, Software Engineering Institute, Pittsburgh, [3] ISO/IEC-15504:1998, Information Technology Software Process Assessment (parts 1-9) International Organization for Standardization, [4] Laporte, C.Y, Renault, A., Desharnais J. M., Habra N., Abou El Fattah, M., Bamba, J. C. (2005) Initiating Software Process Improvement in SMEs: Experiment with CETIC s Micro-Evaluation, in SWED-REK, International Conference on Software Development, Reykjavik, Iceland, University of Iceland, pp ( [5] Mark C. Paulk, Using the software CMM in small Organizations, The SEI, 05/01 [6] Mark Ginsberg and Lauren Quinn, Process Tailoring and the Software Capability Maturity Model, Software Engineering Institute, CMU/SEI-94-TR-024, November 995. [7] Rita Hadden, How Scalable are CMM Key Practices? Crosstalk: The Journal of Defense Software Engineering, April, [8] Donna L. Johnson and Judith G. Brodman, Applying the CMM to Small Organizations and Small Projects, Proceedings of Software Engineering Process Group Conference, March, [9] Terttu Orci and Astrid Laryd, Dynamic CMM for Small Organizations Implementation Aspects, Journal, Stockholm University, April, [10] Habra, N., Niyitugabira, E., Lamblin, A.C. and Renault, A., Software Process Improvement in Small Organizations Using Gradual Evaluation Schema, in Proceedings of the International Conference on Product Focused Software Process Improvement, Oulu, Finland, 1999, [11] N. Habra, A. Renault, S. Alexandre & M. Lopez, OWPL Micro Assessment, in Proceedings for Software Quality Workshop, 24rd International Conference on Software Engineering, ICSE2002, Orlando (FL) [12] Claude Laporte, Alain April and Alain Renault, Applying ISO/IEC Software Engineering Standards in Small Settings: Historical Perspectives and Initial Achievements, Proceedings of SPICE Conference, May 4-5, 2006, Luxembourg [13] Stambollian, Anabel (2006) L amélioration de la performance des processus et pratique logiciels dans des petites entreprises françaises. Project de Synthèse de la maîtrise en génie logiciel, dirige par M. Jean-Marc Desharnais à l école de technologie supérieure. [14] Stambollian, A., Habra N., Laporte C.Y., Desharnais J.M., Renault A., "OWPL: A Light Model & Methodology for Initiating SPI," in Proceedings of the 6th SPICE Conference on Process Assessment and Improvement, Luxembourg, May [15] Wikipedia web site: 5