Software System Documentation Process Maturity Model

Transcription

1 Software System Documentation Process Maturity Model Marcello Visconti, Curtis Cook Computer Science Department, Oregon State University Corvallis, Oregon , USA Abstract Low quality or missing documentation is a major cause of errors in software development and maintenance. We address this problem by proposing a Clevel Documentation Process Maturity model based on the Software Engineering Institute (SEI) Software Development MS turity model. We believe our model will be a useful tool in assessing an organization s documentation level and identifying improvement areas to move the organization to the next level. Keywords: documentation quality, documentation process, process maturity models. 1 Introduction One basic goal of software engineering is to produce the best possible working software along with the best possible supporting documentation. And yet, documentation seems to be considered a second class object and not as important as the software itself. However, empirical data shows that low quality or missing documentation is a major cause of errors in software development and maintenance. In thii paper, we focus on the documentation process and propose a maturity model to address the problem of low quality and/or missing documenta tion. Our approach has been heavily in3uenced by the SEI s software process maturity model. In section 2 we provide empirical evidence of the importance of documentation in both software development and maintenance. In section 3, we introduce our proposed 4level documentation process maturity model. Section 4 de scribes how we plan to validate our maturity model and the next steps in our research. In section 5 we summa rize the main contributions of our model to the software engineering field. Permission to copy without fee all or part of this material is granted provided that the copies are not made or distributed for direct commercial advantage, the ACM copyright notice and the title of the publication and-its date appear,- arid notice is given that ooovina is bv oermission of the Association for Computing Machin&.-To &by otherwise, or to republish, requires a fee and/or specific permission ACM O S/93/0200/0352 $ Importance of Documentation Software quality has been identified as the goal of the 90s in the software engineering field [B&91]. The purpose of this section is to show that software documenta tion products and process are key components of software quality. We will show that poor or missing documentation is a major contributor to the software crisis, namely low product quality and high development and maintenance costs. Documentation is the written record of what the software is supposed to do, what it does, how it does it and how to use it. Virtually everyone agrees that good documentation is important to the analysis, development and maintenance phases of the software process and is an important software product. However, not everyone is aware of how important documentation is. Documentation is probably most crucial to the maintenance phase, which accounts for 60%75% of the total cost of the software [HagerSl, Kell89]. Osborne [Osborne871 reports that documentation accounts for more than 60% of maintenance costs, and is involved in about one-third of the maintenance tasks. A quick understanding of the existing software is a key activity of the maintenance process. Osterweil calls this the knowledge acquisition phase [Gamal88, Chapin88]. Chapin [ChapinBfl asserts that maintenance people spend 40% of their time dealing with documentation. Fjelstad and Hamlen [Fje179] showed that when making a program modification 47% of a maintenance programmer s time is spent studying the program source code and associated documentation. They also found that when correcting errors, the time increases to 62%. Documentation has appropriately been called the castor oil of software process-it is good for you but tastes awful. Far too often documentation may not exist, or if it does exist, it may be incomplete, inaccurate, or out of date. Basili and Rombach [Bomb81 studied an industrial maintenance environment and found that 20% of the maintenance problems are due to bad documentation, with the most frequent problems being documentation faults and documentation clarifications. They claim that better documentation can solve a big percentage of maintenance problems. According to Chapin [Chapin88], maintenance programmers report that for 352

2 most maintenance tasks the source code is the only available documentation. Buckley [Buck891 claims that in most cases maintainers discover that the available documentation is not current, Poston [Post841 asserts that flawed or outdated documentation is more costly than no documentation. Poor quality documentation is a major problem. In a survey of 487 data processing organizations, Lien& and Swanson [Lien811 found documentation quality ranked 3rd in the list of 26 maintenance problem items. They identify documentation quality and adequacy of design specs as accounting for 70% of product quality. Guimaraes [Guim83] claims that the documentation rating has an inverse relationship with the average yearly maintenance expenditures and that maintenance programmers felt that the most important document was an English narrative describing what the programs and modules are supposed to do. Documentation impacts the analysis and development phases as well. Boehm [Boehm75] estimated that documentation costs run about 10% of total Software Development costs. Scheff and Georgon [Scheff91] found that 85% of all software development errors are introduced during requirements, analysis and design. Hamamoorthy [Ramam88] asserts that 80% of software errors in large real-time systems are requirements and design errors due to ambiguity, incompleteness, or faulty assumptions. Basili and Perricone [B&84] conducted a study to analyze the factors that cause errors and found that misunderstanding of a module s specifica tions or requirements constituted the majority of detected errors. Card, Garry and Page [Card873 studied a production environment to evaluate the effectiveness of different technologies and their impact on productivity and reliability. They found that high-use of documentation improves productivity by 11% and reliability by 27% compared to low-use of it. To improve quality, they suggest effective documentation of each phase of development. Fagan [Fagan76] claims that documentation quality inspections are as important as program inspections when the goal is to increase productivity and final software quality. Development impacts maintenance. Hager [Hager891 claims that 60% of the software costs associated with development occur during maintenance. This is be cause development methodologies do not provide adequate visibility for maintenance concerns. Thus ease and cost of maintenance are greatly affected by what takes place during development. He believes that the documentation process plays a key role during development by acting as a design medium that helps organize the information, provides a framework to produce the necessary documents to develop the product, and addresses the need for continuity between the engineering phases through clearly defining document relationships. 3 The System Documentation Process Maturity Model In this section we present a 41evel Documentation Process Maturity model to address the problem of low quality or missing documentation. Our Documentation Process Maturity model is based on both the SE1 Software Process Maturity model and the Capability Maturity model. These models are for the entire software development process and do not specifically address the documentation process. The SE1 model is discussed in [HumphSl]. Each level establishes an intermediate set of goals toward higher levels of process maturity. The new Capability Maturity model identifies key process areas and practices for each level [PaulkSl]. It presents a key-process-area profile instead of a single number indicating a maturity level. Our Documentation Pro cess Maturity model identifies key process improvement areas to move to the next maturity level and produce higher quality documentation. The model The System Documentation Process Maturity Model consists of 4 maturity levels. We present a summary in Table 1. Each proposed maturity level is described as follows: Name Keywords Key Process Areas (to define the basic characteristics) Key Practices (procedures, activities and typical scenarios) Key Indicators (needed in the assessment) Key Challenges (to move to the next level) Key Significance (to software development and maintenance) The concepts key process areas, key practices and key indicators were taken from the SEI s Capability Maturity model whereas the concept key challenge was taken from the SEI s Software Process Maturity model. The rest of the concepts (keywords and key significance) are our own. l Level 1 : Ad-hoc - Keywords: Chaos, variability. 353

3 Ad-hoc Inconsistent Defined Controlled Keywords Chaos Variability Standards Check-off list Inconsistency Product assessment Process assessment Process definition Measurement, Control Feedback Improvement Succinct Documentation Documentation Documentation Documentation Description not a high recognized as recognized as recognized as priority important and important and important and must be done must be done well must be done well consistently Key Process Areas Ad-hoc process Not important Inconsistent application of standards Documentation quality assessment Process definition Process quality assessment and measures Key Practices Documentation Check-off list SQA-like team for Minimum process not used Variable content documentation measures Consistent use of Data collection documentation tools and analysis Extensive use of documentation tools and integration with CASE tools Key Indicators Documentation missing or out of date Standards established SQA-like practices Data analysis Consistent use of and improvement documentation tools mechanisms Key Establish Exercise quality Establish process Automate data Challenges documentation control over measurement collection and analysis standards content Incorporate control Continual striving Specify process over process for optimization Table 1: Documentation Process Maturity Model-Summary Table 354

4 - Key Process Areas : Non-dependable documentation. System documentation process non-existent, ad-hoc or chaotic. Importance of system documentation not understood. - Key Practices : Since documentation may not exist or may be out of date, documentation is not used. No company standards about types of documentation that should or must be created. - Key Indicators : Missing documentation or out of date documentation. Use of code rather than documentation. - Key Challenges : Realization of importance of documentation. Establish required documentation standards. Define what documentation must be created and that it must be kept up to date. 1: Development : The non-dependability of the documentation creates heavy reliance on informal communication creating the conditions for ambiguities, inconsistencies, incompleteness, etc. This results in an environment conducive to errors. * Maintenance : Only dependable documentation is the source code, so program modifications are costly, time-consuming and error-prone. 0 Level 2 : Inconsistent - Keywords: Documentation standards, required documentation, inconsistency - Key Process Areas : Inconsistent use of documentation standards. Poorly controlled documentation process. No consistent use of documentation tools. - Key Practices : Documentation exists, but varies in content and completeness. Checkoff list of required documentation. Use of simple documentation tools (word processors, desktop publishers, editors, spelling checkers, mail, flowcharting, technical drawing) in creation and maintenance of documentation. No quality control over content. - Key Indicators : Documentation standards (IEEE, DOD, local). Documentation tools used. - Key Challenges : Institute quality control over content of documentation. Assess quality of documentation products, Assess utility of documentation products (are they used? If not used, why not?). More effective use of documentation tools. * Development : Standards for which documents must be generated. Since no quality assessment, documentation may be incomplete and may not be up to date. Much informal communication. * Maintenance : Not only the source code will be available but a set of documents generated according to standards. The utility of the documents is suspect, given the lack of control. l Level 3 : Defined - Keywords: Documentation process, product assessment, documentation team. - Key Process Areas : Documentation products quality assessment. Documentation process definition. System documentation team established. - Key Practices : Independent unit established to monitor quality and completeness of documentation (like SQA). Extensive use of documentation tools (document database management, integrated packages), through a defined documentation process. Documentation updated after each change. - Key Indicators : Documentation team goals and procedures. Consistent use of documenta tion tools. Documentation quality assessment methods. - Key Challenges : Establish measures of documentation process quality. * Development : Existence of higher quality documentation will improve the communication process, and attack the causes of errors. In turn, this will affect maintenance, by improving visibility for maintenance concerns. * Maintenance : Extensive use of documentation, allowing maintenance programmer work at a higher level of abstraction, reducing complexity, time and costs. l Level 4 : Controlled - Keywords: Process assessment, measurement, control, feedback, improvement, optimization - Key Process Areas : Documentation process quality assessment and measurement. Data analysis used as feedback into process improvement loop. 355

5 What s - Key Practices : Establishment of minimum measures for documentation process quality. Data collection and analysis. Mechanism to feed measurements into process to identify areas of improvement. Extensive use of documentation tools. Documentation tools integrated with Software CASE tools. - Key Indicators : Data analysis mechanism used to assess the process. Improvementfeedback mechanisms. - Key Challenges : Automate collection and analysis of process data. Maintain a continual optimization of documentation process. Continual striving for improvement incorporated into process. * Development : Existence of a measurable documentation process provides specific indication as to how to proceed making the process more formal and encouraging its reusability. * Maintenance : Overall maintenance costs and time decreased with a well-defined and measurable documentation process. A key aspect in maintenance is having quick access to and understanding of the supporting documentation items relevant to the task at hand. next? The next steps in our research are: l further refinement and validation of 41evel Documentation Process model, l development of an assessment procedure, and l validation of the assessment procedure We expand on these future tasks in the next section. Once this is done, the model is to be used along with the assessment procedure to map an organization s documentation status to one of the maturity levels, generating a documentation process maturity profile. Using this profile, the organization can work on identified areas of improvement to move on to the next level. We intend to use the SEI s assessment procedure and validation experience as a strong starting point in our validation. A complete description of the SEI s efforts can be found in [HumphSl, Humph87J. 4 Validation procedure: Future work In this section we describe the 3-step procedure we are planning on using to validate our model. As a first step to validate the model, we will seek input and feedback from the industry. Our goal is to solicit as much input as possible in defining/refining the current levels that make up our model. We are particularly interested in comments or suggestions about what is in the model, what should not be in the model and what is missing. We have received some industrial input which we have incorporated into our 4level model. Our goal is to obtain additional input from people in industry who deal with this issue on a daily basis. A second step involves developing the assessment procedure itself. We have a draft version of an assessment questionnaire. Our procedure goal is to help identify an organization s documentation process status and areas of immediate improvements. Hence, the organization can concentrate on those areas that are most costeffective in improving its current documentation practices. As a final step, we plan on applying the assessment procedure in the industry to assess the validity of our model to identify an organization s current documenta tion problems and areas of improvement. We strongly believe our model is a viable tool that can be used to attain these objectives, but we need to apply it to a real-world situation to better assess its strengths and/or weaknesses. 5 Final Remarks Incomplete, out of date or missing software document* tion has been recognized as one of the major problems the software development and maintenance community faces. We have proposed a 41evel documentation process maturity model as a vehicle to address the problem of low quality and/or missing documentation. The structure of model is based upon the SEI s Software Process Maturity and Capability Maturity models. SE1 models do not specifically address the documentation process. SE1 Questionnaire includes just a few general questions regarding documentation because SE1 models are not particularly intended to provide a basis for improving documentation practices. Our goal is to fill that void. To do so, for each level we have identified key process areas, practices and indicators as well as key challenges to move to the next higher level. We believe they represent a reasonable spectrum of docu- 356

6 mentation maturity levels for an organization. We also believe our documentation process model can be used to identify problem areas ss well as areas of improvement. The purpose of this paper is to present our model and our plan to validate it. We welcome any comments or suggestions. Acknowledgments Special thanks go to Ken Oar and the software engineers with Hewlett-Packard at Corvallis, Oregon who shared their views on an earlier version of this model. References [Basi84] Basili, Victor R. and Perricone, Barry T. Software errors and complexity: An empirical investigation. Communications of the ACM, Vol. 27, No. 1, [BasiSl] Basili, Victor R. and Musa, John D. The future engineering of software: a management perspective. IEEE Computer, September, [Boehm75] Boehm, Barry W. The high cost of software. From Horowitz, Practical Strategies for Developing Large Software Systems, Addison- Wesley, Reading, Mass, [Buck891 Buckley, J. Some standards for software maintenance. Standards, IEEE Computer, November [Card871 Card, David N., MC Garry, Frank E. and Page, Gerald T. Evaluating software engineering technologies. IEEE Transactions on software engineering, Vol. SE13, No. 7, [Chapin87] Ch a p in, Ned. The job of software maintenance. Proceedings Conference on Software Maintenance, IEEE, [Chapin Chapin, Ned. Software maintenance life cycle. Proceedings Conference on Software Maintenance, IEEE, [Fagan76] Fagan, M.E. Design and code inspections to reduce errors in program development. IBM Systems Journal, Vol. 15, No. 3, [Fje179] Fjelstad, R.K and Hamlen, W.T. Application program maintenance study report to our respondents. Proceedings GUIDE 48, Philadelphia, PA, [Gama188] Gamalel-Din, Shehab A. and Osterweil, Leon J. New perspectives on software maintenance processes. Proceedings Conference on Software Maintenance, IEEE, [Guim83] Guimaraes, Tor. Managing application program maintenance expenditures. Communications of the ACM, Vol. 26, No. 10, [Hager891 Hager, James A. Software cost reduction methods in practice. IEEE Transactions on software engineering, Vol. SE15, No. 12, [Hager911 Hager, James A. Software cost reduction methods in practice: a post mortem analysis. Journal of systems and software, Vol. 14, No. 2, [Humph87] Humphrey, Watt S. and Sweet, W. L. A method for assessing the software engineering capability of contractors. CMU/SEI-87-TR-23, ESD-TR , September [HumphSl] Humphrey, Watt S., Snyder, Terry R. and Willis, Ronald R. Software process improvement at Hughes Aircraft. IEEE Software, July [Ke1189] Kellner, Marc I. Non-traditional perspectives on software maintenance. Panel, Proceedings Conference on Software Maintenance, IEEE, [Lien811 Lientz, Bennet P. and Swanson, E. Burton. Problems in application software maintenance. Communications of the ACM, Vol. 24, No. 11, [Osborne871 Osborne, Wilma M. Building and Sustaining Software Maintainability. Proceedings Conference on Software Maintenance, IEEE, [PaulkSl] Paulk, M. C., Curtis, B., Chrissis, M. B., et al. Capability Maturity model for software. CMU/SEI-91-TR-24,.August [Post841 Poston, Robert M. When does more documentation mean less work?. Software Standards, IEEE Software, October [Ramam88] Ramamoorthy, C.V. Our job is to reduce the errors. From Myers, Ware. Can software for the strategic defense initiative ever be error free? IEEE Computer, Vol. 21, No. 11, [Romb87] Rombach, H. Dieter and Bssili, Victor R. Quantitative assessment of maintenance: an industrial case study. Proceedings Conference on Software Maintenance, IEEE, [SchefHJl] Scheff, Benson H. and Georgon, Thomas. Using documentation blueprints to produce mandated DOD data items. Journal of Systems and Software, Vol. 14, No. 2,