MPS.BR: A Tale of Software Process Improvement and Performance Results in the Brazilian Software Industry

Transcription

1 MPS.BR: A Tale of Software Process Improvement and Performance Results in the Brazilian Software Industry Gleison Santos UNIRIO Federal University of Rio de Janeiro State ZIP Av. Pasteur, 458 Rio de Janeiro, Brazil gleison.santos@uniriotec.br Marcos Kalinowski, Ana Regina Rocha, Guilherme Horta Travassos COPPE/UFRJ Federal University of Rio de Janeiro POBOX ZIP Rio de Janeiro, Brazil mkali, darocha, ght@cos.ufrj.br Kival Chaves Weber, José Antonio Antonioni SOFTEX Association for Promoting the Brazilian Software Excellence POBOX 6123 ZIP Campinas, Brazil kival.weber, jaa@nac.softex.br Abstract In December 2003 a Brazilian nationwide program was created aiming at improving software processes. The MPS.BR Program is responsible for a great evolution in the software development scenario in Brazil by providing the means to support software process improvement initiatives based on the MPS Model. In 2008 a project named imps was launched to enable the analysis of the Performance Results of organizations that adopted the MPS Model. This paper describes some of the findings of the studies that have already been conducted. Preliminary evidences show improvement trends regarding cost, quality, schedule, and productivity, especially for those organizations that adopted the MPS Model for continuous improvement purposes. Keywords: Software Process Improvement, Performance Results, Return on Investment I. INTRODUCTION Good software engineering must always include a strategy for producing quality software [1]. From 1993 there is a strategy for improving software quality in the Brazilian Software Industry both the quality of the software product and processes [2]. The MPS.BR Program is a nationwide mobilization program [3, 4] aiming at improving software process in Brazilian organizations both large companies and small and medium-size enterprises (SME). Thus, it was created the MPS Model. The MPS.BR program (acronym of the Portuguese expression Melhoria de Processo do Software Brasileiro or Brazilian Software Process Improvement) was created in December 2003 by the Association for Promoting the Brazilian Software Excellence (SOFTEX), a private notfor-profit organization aiming at promoting Brazilian software industry competitiveness. SOFTEX has a network of 22 agents (in 20 cities of 12 Brazilian states). There are over 1,600 SOFTEX affiliated companies 70% are SMEs (see In the long-term planning, the MPS.BR program is aligned to SOFTEX s objective of development of the Brazilian software market and the sustainable increase of the Brazilian Software and Services Industry competitiveness. By May 2010, 217 organizations have already successfully been appraised on the MPS model. Over 70% are SMEs. Nevertheless, it is important to demonstrate the benefits reached by such organizations after the MPS model adoption. This way more organizations can be influenced to adopt it and the ones already engaged in software process improvement initiatives based on the MPS model can be motivated to continue their efforts to improve their software process and product quality. This paper describes the MPS.BR Software Process Improvement (SPI) and Performance Results. Section 2 discusses the background in SPI. Section 3 presents SPI based on the MPS model and the MPS.BR program. Section 4 presents the Performance Results of organizations that adopted the MPS model. Section 5 presents our final considerations. II. BACKGROUND Over 70% of the Brazilian software industry is constituted of SMEs and since 2004 few organizations had adopted reference models [5]. A study found that usually organizations only implement good software engineering practices if demanded by processes appraisals and assessments [6]. It is a general belief that the use of such good practices may improve the performance of organizations related to costs, schedule, productivity, quality, customer satisfaction and return on investment [7]. Different approaches have been proposed to improve and assess software development capabilities motivated by the need to have better techniques to select software suppliers [8]. Software process models such as ISO/IEC [9], ISO/IEC

2 15504 [10], and CMMI-DEV [11] describe what to implement but not how to implement. Studies have been conducted to evaluate the costs, benefits and impacts of SPI initiatives based on CMMI-DEV and ISO/IEC [7, 12]. Some authors have also investigated the difficulties organizations have when adopting such models [13, 14]. These studies help to understand the difficulties on adopting CMMI- DEV and ISO standards in software organizations, especially SMEs. Staples et al. [15] report that SMEs recognize the SPI efforts benefits but do not consider feasible the deployment of a model such as CMMI-DEV. Wangenheim et al. [14]. and Cater-Steel et al. [16] also report that one of the most important challenges regarding SMEs is to show to them the expected business results. The authors recognize the need in reducing process assessment costs and the amount of time necessary to make the SPI benefits visible. According to Staples et al. [15], SPI approaches should focus on SMEs, have low costs and short implementation time. III. SOFTWARE PROCESS IMPROVEMENT: MPS MODEL AND MPS.BR PROGRAM From December 2003 many activities and actions were executed by SOFTEX to establish and sustain the MPS.BR program with its community of MPS model practitioners [17]. This constituted the MPS Implantation Phase ( ). The current Consolidation Phase ( ) aims to consolidate the MPS.BR program structure, to improve the MPS model, and disseminate the MPS model in Brazil. The MPS model is constituted of three main components: MPS Reference Model (MR-MPS); MPS Assessment Method (MA-MPS); and MPS Business Model (MN-MPS). These components where established during the Implantation Phase and are continually evolving due to the experiences gained along the time. MR-MPS and MA-MPS are conformant with ISO/IEC [9] and ISO/IEC [10], CMMI-DEV [11] compatible, based on software engineering best practices, and in accordance with the Brazilian companies reality. The MPS Reference Model (MR-MPS) is documented in the form of three guides (available at The MPS General Guide provides a general definition of the MPS model and common definitions to all other guides. The MPS Acquisition Guide describes an acquisition process for software and related services. The MPS Implementation Guide provides technical guidance for implementing the MR-MPS. Besides, it describes the theoretical concepts to ground the processes defined in the MR-MPS maturity levels; establishes the means to give uniformity to Software Engineering knowledge among Software Process Improvement (SPI) practitioners and reducing the risks of misunderstanding the implementation issues essential for successfully satisfying the MR-MPS processes outcomes. Table I shows the structure of the MR-MPS maturity levels (ML) which are defined in two dimensions (based on the requirements of ISO/IEC 15504): process dimension and process capability dimension (process attributes). CMMI is SEI/CMU trademark. MPS.BR, MR-MPS, MA-MPS, MN-MPS, and imps are SOFTEX service marks. TABLE I. MR-MPS MATURITY LEVELS (ML) STRUCTURE ML Processes Process Attributes A (no new processes are added) 1.1, 2.1, 2.2, 3.1, 3.2, 4.1*, 4.2*, 5.1*, 5.2* B Project Management (new outcomes) 1.1, 2.1, 2.2, 3.1, 3.2, 4.1*, 4.2* C Decision Management, Risk Management, and 1.1, 2.1, 2.2, 3.1, Development for Reuse D Requirements Development, Product Design and Construction, Product Integration, Verification, and Validation E Human Resources Management, Process Establishment, Process Assessment and Improvement, Project Management (new outcomes), and Reuse Management F Measurement, Configuration Management, Acquisition, Quality Assurance, and Project Portfolio Management , 2.1, 2.2, 3.1, , 2.1, 2.2, 3.1, , 2.1, 2.2 G Requirements Management and Project 1.1, 2.1 Management * These Process Attributes (PAs) are applicable only on selected processes. The others PAs must be applied on all processes. The initial MR-MPS maturity level is the level G constituted of the two most critical software processes to SMEs: Requirements Management and Project Management. Levels A and B are the highest MR-MPS maturity levels focusing on continuous process improvement. The first version of the MR-MPS was defined in 2005 and the current one in 2009 with improvements such as: (i) revision of PAs to improve the adherence to ISO/IEC [10]; (i) general review to improve compatibility with ISO/IEC [9] and ISO/IEC [10]; (ii) definition of the Project Portfolio Management process in the MR-MPS Level F, based on ISO/IEC [9], to guarantee that companies analyze their project portfolios to initiate and sustain necessary, sufficient, and suitable projects in order to meet the strategic objectives of the organization; (iii) MPS Implementation Guide review to improve the understanding of the outcomes described in the MPS General Guide and to clarify what is suggested and what is mandatory in MA-MPS assessments; (iv) review of MPS Implementation and Acquisition Guides due to comments and experiences collected from MPS assessors (i. e., persons authorized to lead or participate in official MPS assessments) and MPS model implementation consultants during the last two years in order to facilitate the MPS model interpretation and, not less important, to make the model more suitable to the organization reality. The MPS Assessment Guide describes the requirements for accreditation of organizations to provide MPS assessment services (MPS Assessment Institutions a.k.a. AI), MPS Competent Assessors and MPS Provisional Assessors, and also describes the assessment process defined to support the application of the MA-MPS method. This method objective is to verify the maturity of an organization unit in the execution of its software processes. All MPS assessments are audited to increase their credibility and are valid for only three years. In the same direction, a new version of the MA-MPS method was also released in 2009 based on experiences collected and to make the model more suitable to the reality of

3 Brazilian organizations. The conformity with ISO/IEC [10] was also reviewed and increased. In order to guarantee the success of the MPS model, it is essential that the organizations can effectively adopt it and obtain benefits from implementing software process improvement. In this way, the MPS Business Model (MN- MPS) was defined and has a very important role in the management of the MPS.BR program, describing all the business rules. It aims at improving commitment of all MN- MPS participants to foster continually the success of the program by supporting the relation defined as triangle of services [18] among the Enterprise (the MPS.BR program itself), the Collaborators of the Front Line (such as those responsible to support the MPS implementations or to conduct the MPS assessments, and the MPS instructors), and the Client (the organizations that adopted the MPS model). In order to manage the MPS.BR program itself, an organizational structure was defined. The MPS.BR program structure units are: (i) MPS.BR Program Team: responsible to manage the program activities; (ii) MPS Technical Model Team: responsible to develop and maintain the MPS model, and to prepare and execute MPS model trainings; and (iii) MPS Accreditation Forum: responsible to accredit organizations to provide MPS model-based implementation and assessments services, to evaluate and control implementation and assessment results, and to ensure that these accredited organizations execute their activities within expected ethical and quality limits. In the current phase, the MPS.BR program basically maintains its same major original challenges: consolidation of the MPS.BR program structure and MPS model improvement, with four goals in the period : (i) consolidation of the MPS.BR program structure; (ii) MPS model improvement; (iii) accreditation of Implementation Institutions (II) and Assessment Institutions (AI); (iv) analysis of MPS.BR Lessons Learned [19] and Performance Results [20]; dissemination of the MPS model in Brazil, with four goals in the period : (i) training regarding the MPS model: courses, exams and workshops; (ii) support to groups of enterprises for MPS based implementation and assessment initially focused on the levels G and F (basis of the pyramid), under agreement with the Organizing Institution of Groups of Enterprises (a.k.a IOGE, acronym of the Portuguese expression Instituição Organizadora de Grupos de Empresas ); (iii) support to groups of enterprises for MPS based implementation and assessment in the levels E, D and C (middle of the pyramid), under agreement with the IOGEs; (iv) dissemination of the MPS software acquisition process. With the growing number of Brazilian organizations adopting MPS, it had sprouted the interest on qualitatively understanding the performance variation of them, concerning with measures such as time, productivity, cost, and quality due the MPS model adoption. With this purpose, the imps project (information to monitor and provide evidence regarding performance variation of software organizations that adopted the MPS model) was started. The main goal of this project is to periodically monitor and provide evidence, based on surveys, regarding the performance of software organizations that adopted the MPS model [20]. A summary of the 2009 imps project results is described hereafter. IV. IMPS: PERFORMANCE RESULTS OF SOFTWARE ORGANIZATIONS THAT ADOPTED THE MPS MODEL The 2009 imps results will be presented under two perspectives: 2009 characterization and 2008/2009 variation analysis. The goal of the characterization is to outline the performance of organizations that adopted the MPS model in 2009 while the 2008/2009 variation analysis observes performance variation of organizations that adopted the MPS model. The following subsections describe, respectively, the results of the 2009 characterization and the 2008/2009 performance variation analysis. The results of the 2008 baseline characterization can be found in [21]. A Characterization Analysis The characterization analysis aims to outline the performance of the organizations that adopted the MPS model in questionnaires from different organizations (20 starting the MPS implementation, 25 in assessment process, 57 assessed MPS level G, 26 assessed MPS level F, and 7 assessed MPS levels E-A) were analyzed. Given that most organizations are still in the initial maturity levels, they were grouped during analysis in the following four categories: organizations starting to implement the MPS model, organizations assessed MPS level G, organizations assessed MPS level F, and organizations assessed MPS levels E-A. The obtained results allowed different interpretations, and it is possible that some confounding and even political and economic factors of the year 2009 may have influenced the data. However, some behaviors (that we believe are related to the MPS model adoption ) could be observed [22]. a) Customer Satisfaction It can be observed that customer satisfaction has been reported higher for organizations that adopted the MPS Model. Considering organizations starting the implementation, 50% reported to have fully or largely satisfied customers. Among those that adopted MPS this number rises to 68.13%. Considering only the organizations between levels E-A customer satisfaction reaches 71.43%. b) Satisfaction with the MPS Model Regarding satisfaction with the MPS Model, 71.11% (96 organizations) reported being completely satisfied with the model and 27.41% reported being partially satisfied. Only 0.74% (1 organization) reported being not satisfied and 0.74% reported not knowing its level of satisfaction yet. All companies with maturity level F or higher declared to be fully or partially satisfied. c) Other Maturity Models Among other maturity models and standards, the most used by the organizations is CMMI-DEV. The combination MPS

4 and CMMI can be observed with more frequency. Considering the organizations starting the implementation, 10% have some CMMI maturity level. At level G the percentage of organizations with CMMI maturity levels is 10.53%. At level F, this number rises to 11.54% and between levels E-A it reaches 57.14%. d) Project Size Regarding project size, 44 (33.9%) of the 135 surveyed organizations reported measuring the size of their projects in function points, which was the most used measure of size, followed by use case points, used by 20 organizations (14.81%). Figure 1 shows the median of the average project size of organizations that use function points, for each grouping used in the study. Figure 1. Median Project Size in Function Points. While the median for organizations starting the implementation is 200 function points, the median for companies in the levels of E-A is 260. There is a positive correlation between the increase in median and the increase of maturity level of e) Productivity Productivity appears higher for organizations that adopted the MPS model. The highest median was in the group of level G organizations. However, it is important to note that productivity should not be observed in isolation, since it may vary for different project types and different quality and cost expectations. Additionally, productivity takes into account other base measures, which, as discussed previously, may be more reliable for organizations with maturity levels F or higher, which have an institutionalized measurement process. Figure 2 shows the median representing productivity of projects of organizations that use function points for each grouping used in the study. While the median productivity for organizations starting the implementation is 40 function points per month, the median for organizations in the levels E-A is There is a positive correlation (+0.46) between the increase of the median and the MPS maturity level. Additional investigation is necessary to understand what it could mean. Further details on the 2009 characterization can be found in [22], the following sub-section presents the 2008/2009 performance variation of organizations that adopted the MPS Model. Figure 2. Median Productivity (Function Points/Month). B. 2008/2009 Performance Variation Analysis The indicators for this analysis were those defined in the imps study plan: A. Net Sales Variation, B. Number of Customers in Brazil, C. Number of Employees, D. Average Project Cost, E. Average Project Duration, F. Average Project Size, G. Productivity, H. Quality, I. Customer Satisfaction, and J. Return on Investment (ROI). At all, 43 organizations, with a questionnaire for the year 2008 and another for the year 2009, were considered and grouped according to the same criteria used in the characterization: level G (22), level F (17), and levels E-A (4). Moreover, a new group with 9 organizations was analyzed, concerning those organizations that have increased or revalidated their maturity level in this period and answered the questionnaire. As defined, imps data is always collected in order to avoid competitive comparison between organizations. To observe the performance variation behavior for each measure, the relative percentage of organizations (based on the number of valid answers) with increasing, decreasing or not change has been used. The overall results present interesting trends especially for those 9 organizations that changed (got higher) or revalidated their MPS maturity level, as shown in Figure 3. The main characteristic of these organizations, regardless the level they were assessed, refers to MPS adoption and continuity in developing software improvements following MPS model s guidelines. It is possible to observe that, according to the data provided by the organizations, the indicators show consistent behavior with the assumptions regarding the use of software development processes combined with good software engineering practices. For instance, some cost and duration reduction trends can be observed in combination with the increasing of quality and productivity. We believe that this combination of events may be influencing positively the other indicators for these organizations, related to the increase of net sales, number of customers, employees, customer satisfaction, and ROI. Further investigation needs to be performed in order to identify and mitigate, possible threats of validity regarding these results. Further details and analyses on the 2008/2009 performance variation, including separate results of each MPS Level grouping can be found in [22].

5 Indicator A B C D E F G H I J Valid Answers Conf. Level(%) Figure 3. Performance Variation of Organizations that adopted the MPS and increased/revalidated their maturity level. V. CONCLUSIONS Until May 2010, 217 organizations (139 MPS-G, 61 MPS- F, 6 MPS-E, 1 MPS-D, 4 MPS-C, 0 MPS-B, and 6 MPS-A) had gone through successful MPS model-based assessments in Brazil. Over 70% are SMEs. So far, 4,068 SPI practitioners attended MPS model courses in different cities and 1,230 were successfully approved in the MPS model exams. Another important strategy to guarantee the adoption of the MPS model is to accredit institutions in different parts of the country to provide MPS model-based services: 18 IIs are accredited to provide MPS model-based implementation services and 13 AIs are accredited to provide MPS model-based assessment services. Moreover, 120 MPS model implementation consultants are associated to accredited IIs effectively working on MPS model-based initiatives and 68 competent and provisional assessors are associated to accredited AIs with conditions to conduct MPS model-based assessments. Concerning the MPS.BR program, we come to the following conclusions: (i) until now, the program sustainability was guaranteed and the achieved results have been excellent, outperforming the expected results; (ii) it is an important business for the Brazilian software industry, with a strong interaction among University, Industry and Government; (iii) the MPS.BR is a long-term program; (iv) the MPS.BR is a nationwide mobilization program; (v) it represents a great effort to supporting the development capacity in Brazil (of individuals, institutions and organizations interested in the MPS model) to continually improve both the software processes and innovation in the software industry for competitive advantage in local and global markets. Moreover, due to the initial imps results, we can indicate the MPS.BR program importance to the Brazilian software organizations. Regarding the 2009 characterization, it was possible to observe that organizations that adopted the MPS model reported higher customer satisfaction were able to handle bigger projects, present lower time estimation errors, and show themselves more productive, when compared to organizations starting the MPS model implementation. Besides, the satisfaction with the model indicates that more than 98% of the organizations reporting to be partially or fully satisfied with the model. Regarding the 2008/2009 variation analysis, for companies that are using the MPS model, it was possible to observe that, independent of the maturity level, the MPS adoption may have contributed to increase the number of clients, the net sales and the number of employees, without affecting customer satisfaction [22]. In general, organizations reported return on investment and, especially for those organizations that evolved or internalized the MPS model, improvement trends could be observed regarding cost, quality, time, and productivity, basic assumptions when developing software according to engineering precepts. REFERENCES [1] S. L. Pfleeger, Software Engineering: theory and practice, 2nd ed.: Prentice-Hall, [2] K. C. Weber and M. Pinheiro, "Software Quality in Brazil", Quality World Magazine, vol. 21, November [3] K. C. Weber, A. R. Rocha, A. C. Rouiller, A. Crespo, A. Alves, A. Ayala, A. Gonçalves, B. Paret, C. Vargas, C. Salviano, C. Machado, D. Scalet, D. Petit, E. Araújo, J. C. Maldonado, K. Oliveira, L. C. Oliveira, M. Girão, M. Amaral, R. Campelo, and T. Maciel, "Uma Estratégia para Melhoria de Processo de Software nas Empresas Brasileiras", in Proceedings of QUATIC 2004 (5th International Conference for Quality in Information and Communications Technology) Portucalense University, Porto, Portugal, [4] A. R. Rocha, M. Montoni, K. C. Weber, and E. E. R. d. Araújo, "A Nationwide Program for Software Process Improvement in Brazil", in

6 Proceedings of the 6th International Conference on Quality of Information and Communications Technology: IEEE Computer Society, [5] MCT, "Ministério da Ciência e Tecnologia", Brasília, Brazil, in accessed in 01/05/2008. [6] M. O. Nogueira, Qualidade no Setor de Software Brasileiro, D. Sc., COPPE, UFRJ, Rio de Janeiro, Brasil, [7] D. R. Goldenson and D. L. Gibson, "Demonstrating the Impact and Benefits of CMMI: An Update and Preliminary Results", in SEI Special Report, CMU/SEI-2003-SR-009, [8] T. P. Rout, K. E. Emam, M. Fusani, D. Goldenson, and H.-W. Jung, "SPICE in retrospect: Developing a standard for process assessment", Journal of Systems and Software, vol. 80, pp , [9] ISO/IEC, "Systems and software engineering Software life cycle processes". vol. ISO/IEC 12207:2008: The International Organization for Standardization and the International Electrotechnical Commission, [10] ISO/IEC, "Information Technology Software Process Assessment". vol. ISO/IEC 15504: Parts 1-9, The International Organization for Standardization and the International Electrotechnical Commission, [11] M. B. Chrissis, M. Konrad, and S. Shrum, CMMI (Second Edition): Guidelines for Process Integration and Product Improvement: Addison Wesley Professional, [12] A. Tuffley, B. Grove, and G. McNair, "SPICE for small organisations", Software Process: Improvement and Practice, vol. 9, pp , [13] F. G. Wilkie, D. McFall, and F. McCaffery, "An evaluation of CMMI process areas for small- to medium-sized software development organisations", Software Process: Improvement and Practice, vol. 10, pp , [14] C. Gresse von Wangenheim, T. Varkoi, and C. F. Salviano, "Standard based software process assessments in small companies", Software Process: Improvement and Practice, vol. 11, pp , [15] M. Staples, M. Niazi, R. Jeffery, A. Abrahams, P. Byatt, and R. Murphy, "An exploratory study of why organizations do not adopt CMMI", Journal of Systems and Software, vol. 80, pp , [16] A. Cater-Steel, M. Toleman, and T. Rout, "Process improvement for small firms: An evaluation of the RAPID assessment-based method", Information and Software Technology, vol. 48, pp , [17] M. A. Montoni, A. R. Rocha, and K. C. Weber, "MPS.BR: a successful program for software process improvement in Brazil", Software Process: Improvement and Practice, vol. 14, pp , [18] J. Teboul, Serviços em cena: o diferencial que agrega valor ao seu negócio. Brasília, Brazil: IEL/NC, [19] A. R. Rocha and K. C. Weber, MPS.BR Lessons Learned. Campinas, Brazil: SOFTEX, 2008, available at [20] M. Kalinowski, K. C. Weber, and G. H. Travassos, "imps: An Experimentation Based Investigation of a Nationwide Software Development Reference Model", in 2nd International Symposium on Empirical Software Engineering and Measurement (ESEM 2008), Kaiserslautern, Germany, [21] G. H. Travassos and M. Kalinowski, imps: Resultados de Desempenho de Organizações que Adotaram o MPS (in Portuguese). Campinas, Brazil: SOFTEX, 2008, available at [22] G. H. Travassos and M. Kalinowski, imps 2009: Characterization and Performance Variation of Software Organizations that Adopted the MPS Model. Campinas, Brazil: SOFTEX, 2009, available at