Achieving SA-CMM Maturity Level A DoD Acquisition Management Experience

Transcription

1 Achieving SA-CMM Maturity Level A DoD Acquisition Management Experience Michael Markarian 37 Penkivil Street, Willoughby, NSW 2068 Michael.Markarian@erols.com Matthew Fisher Software Engineering Institute Carnegie Mellon University 4500 Fifth Ave. Pittsburgh, PA USA mjf@sei.cmu.edu Abstract. For many years now, United States federal government agencies have seen the advantages in outsourcing software engineering support for both new product development and maintenance. This approach provides federal government agencies access to a wide variety of skilled professionals that have the essential knowledge and experience relative to current technology, and meet the requirements for developing a solution. Although outsourcing alleviates management oversight at a software engineering level, there is still a need for management oversight of a software project. Software acquisition management has been established as a common practice within government agencies, as a result of the shift to outsourcing software engineering support. Active involvement by the acquisition management organization is still required, but at an oversight level. Therefore, an Acquisition Management Organization (AMO) must have sound practices in place to effectively manage the software acquisition. In this paper we discuss an Acquisition Management Organization that is a product management office with a mission to maintain existing computer information systems. The importance of having sound software acquisition management practices in place was recognized, and the AMO embarked on a mission to achieve this level of excellence. The Software Engineering Institute (SEI) in conjunction with the acquisition management community developed a software acquisition model that provided the foundation for assessing the effectiveness of any Acquisition Management Organization. The model, known as the Software Acquisition Capability Maturity Model 1 (SA-CMM ) is based on the fundamentals of software acquisition processes and practices. The model provides an objective assessment of an organization s capability and maturity in software acquisition management, and forms the basis for establishing a process improvement program to achieve a maturity level. This paper provides an overview on the experiences of applying SA-CMM on a federal government agency, to establish more effective acquisition management processes and practices. The life cycle of this experience includes the planning stages of identifying the model to use, through to completion of the final assessment where a maturity level 2 rating was achieved against the SA-CMM (version 1.03) model. It also provides an indication of obstacles that could potentially be expected, and how the organization resolved these obstacles to better achieve its mission and benefit from its process improvement effort. 1. BACKGROUND As we all know software is pervasive in many of our systems. Software provides a critical means to achieve much of the functionality of these modern systems. The term software-intensive system has been used to reflect this dependence and the growing demand for such systems, whether they are financial, automotive, or weapon systems. Typically, this demand is being satisfied by acquiring the system and attendant software rather than developing or building the system in house. 1 Software Capability Maturity Model, SW-CMM, and Software Acquisition Capability Model, SA-CMM are registered in the U.S. Patent and Trademark Office.

2 However, there have been and are widespread problems within projects acquiring softwareintensive systems. Cost overruns, schedule slips, and poor system/software performances are typical symptoms of such problems. Studies 2 have indicated that many of these problems reside with the management of the acquisition; and, specifically with the lack of discipline and the poor process capability and process maturity of the acquirer. All too often, software-intensive system acquirers place the blame of poor-quality software on the suppliers (developers). Acquirers and suppliers are on different sides of the same system. They can engage in mutually beneficial behaviors or mutually destructive behaviors. A poor acquirer can inhibit a good supplier. The bottom line is that the acquirer s management processes and practices and resultant decisions can negatively impact the software development process of the supplier. Figure 1 gives us some indication why this situation exists. We are seeing more and more that the mismatch of the acquirer and the supplier as far as their process capability and maturity, and level of experience are contributing to many of the problems Supplier (developer) mentioned earlier. If delivery of quality software products and services is a joint responsibility of the acquirer and the supplier, then we need to improve the practices of both. Figure 1: Acquirer/Supplier Capability Mismatch Let s first consider some noticeable symptoms such as those shown in bottom left quadrant of Figure 1. The most common is the ability for effective communications across all the components of an acquisition management organization. Without regular communication of activities, events, work in progress and team participation, the underlying goals may not be accomplished and can lead to further degradation in other areas of the AMO. Such areas include the availability of production systems that are maintained by the AMO. Any unscheduled downtime of production systems can be devastating to an organization s business objectives if the computer 2 Acquirer capability/maturity Management Capability Level Mismatch mature buyer must mentor low maturity developer outcome not predictable Disaster constant crises no req s mgt. no risk mgt. no discipline no process... no product capability/maturity Matched Team match of skills, maturity team risk approach execution to plan measurable performance quantitative management highest probability of success Mismatch Customer is always right hurts. Customer encourages short cuts. Department of Defense, Defense Science Board Report Acquiring Defense Software Commercially, June 1994 system is unavailable for daily business processing. There is a saying that all roads lead to requirements. Having partial or no requirements management process in place can provide an element of chaos and poor guidance for the work to be accomplished by all stakeholders during the acquisition. A software supplier will attempt to fulfill the requirements to the best of their ability. But unless the AMO office has clearly defined what those requirements are, there is no guarantee that the true needs of the business will be fulfilled. A clearly defined set of requirements sets the direction of the acquisition, and ultimate accomplishment in satisfying the needs of the organization s business objectives. Having well developed and institutionalized processes, necessary practices, training and tools supporting the AMO will guide the AMO from a crisis management environment into a more structured and disciplined environment. This same concept of establishing a disciplined approach also applies to the supplier, so that software supplier receiving guidance from the AMO has the ability to deliver the quality products and services as expected. 2. THE SOFTWARE ACQUISITION CAPABILITY MATURITY MODEL (SA-CMM) 2.1 Evolution of CMM The Software Acquisition Capability Maturity Model (SA-CMM ) [1] was developed to help improve the acquirer's ability to manage acquisitions by providing a mechanism to discipline the acquirer's software acquisition processes. It was developed to address the common problems noted above. The SA- CMM focuses on how the acquisition organization manages its internal business, not on how the supplier manages it s development project. From another perspective, the Software Capability Maturity Model (SW-CMM ) [4] and the Capability Maturity Model Integrated (CMMI ) [2] describes the supplier's role while the SA-CMM describes the acquirer's role in the acquisition process. Organizations have used the SA-CMM to instill discipline not only in their software acquisition processes, but also in their general acquisition processes. This is possible because of the flexibility and adaptability of the SA-CMM to apply to each organization's unique business needs [3]. If applied correctly, the SA-CMM results in an introspective view of an organization's ability to accomplish its acquisition mission. Typically, such introspection reveals areas for improvement that include: the organization lacks institutionalized acquisition processes, the processes are inefficient and sometimes ineffective; organizational overlaps exit; responsibilities are not well defined; and visibility into projects is poor. Application of SA- CMM helps an organization understand the existence of these conditions. This realization provides the

3 basis for developing process improvement plans. 2.2 SA-CMM Architecture The architecture of the SA-CMM is shown in Figure 2. The architecture is structured into five levels of Level Focus Key Process Areas 5 Optimizing 4 Quantitative 3 Defined 2 Repeatable 1 Initial Continuous process improvement Quantitative management Process standardization Basic project management Competent people and heroics Acquisition Innovation Management Continuous Process Improvement Quantitative Acquisition Management Quantitative Process Management Training Program Management Acquisition Risk Management Contract Performance Management Project Performance Management User requirements Process Definition and Maintenance Transition to Support Evaluation Contract Tracking and Oversight Project Management Requirements Development and Mgt. Solicitation Software Acquisition Planning increasing process maturity. Each maturity level (except Level 1) contains key process areas (KPAs), which are clusters of important, related practices. Figure 2: SA-CMM Architecture Note that the primary focus of Level 2 is basic project management within a single project. The Software Acquisition Planning and Solicitation KPAs provide the acquisition plans and requirements for the solicitation package and the resulting contract. Requirements for the solicitation and contract area developed under the Requirements Development and Management KPA (These requirements are passed to the supplier through the solicitation package and contract). Although the acquisition organization develops and manages the requirements from the beginning of the project, both the acquirer and the supplier have roles in managing requirement changes throughout the contract period of performance [3]. 3. THE SOFTWARE ACQUISITION MANAGEMENT ORGANIZATION Higher Quality Productivity Lower Risk Higher Risk Rework 3.1 Characteristics of the Acquisition Management Organization The AMO for this process improvement effort is a DoD office primarily responsible for the development and maintenance of computer systems for a large user base. This AMO does not perform any in-house software development, but acquires both products from vendors and services from suppliers. The AMO has established Vision and Mission statements that are it s guiding principles. Planning and activities are determined by the needs of the user community. This ranges from shortterm needs where modifications are needed to existing systems based on requirements submitted from the user community, to long-term needs which is combined input from the user needs based on projected needs and available technology that could be used to enhance system performance and functionality. Some specific responsibilities of the AMO include the development of requirements that translates the user needs into a common set of specifications. An acquisition strategy and plan is developed documenting the approach to be taken and the work to be accomplished during the acquisition. A solicitation package is developed from the requirements for soliciting bids from vendors that are fully qualified to perform the work. Preparation of a contract for supplier services requires both financial and legal competence to ensure viable source selection and correct expenditure of funds. A government procurement or contract organization services this AMO for any support related to contract administration. In addition to acquisition management responsibilities, technical and project management skills are also pertinent to the deliver of quality software products and services. These are traditionally overlooked, and reliance is placed on the supplier to provide input. In fact, these responsibilities are precisely those embodied in the SA-CMM KPAs at level 2 as shown in Figure 2, i.e., basic acquisition project management. The key stakeholders supporting the AMO include the user community, the procurement or contract office and the software supplier that provides software solutions. 4. THE PROCESS IMPROVEMENT EFFORT 4.1 The Process Improvement Strategy The first step in the effort for the AMO was to determine the most appropriate CMM model to use. The AMO selected SA-CMM to help identify and understand the areas for improvement, and used this information as the basis for process improvement planning (Caution should be taken when considering applying the Software Subcontract Management KPA from the SW-CMM to an AMO. This KPA is limited to subcontract management for a software development organization and does not provide the foundation for managing an acquisition.). In addition, the process improvement effort of the AMO required the software supplier to also establish a disciplined environment using the SW-CMM. This integrated approach would help ensure that the overall combined work of the AMO and supplier team would produce quality products and services. This requirement led the software supplier to establish the necessary processes, practices and tools based on the guidelines provided by the SW-CMM. With the combination of using the SA-CMM model for the AMO office and the SW-CMM model for the software supplier, this approach (shown in Figure 3) would guide both the AMO and the software supplier the capability to operate in the desired quadrant of Figure 1. Although there are a total of 5 maturity levels in the SA-CMM model, only level 2 key process areas were selected for this initial process improvement effort. As both the AMO office and software supplier processes mature, then the next level key process areas may be considered. The goal is to step up the maturity level

4 and not take leaps which would only prove disastrous, Information Requirements of the User Community AMO (Acquirer) Software Contractor (Supplier) RDM ACQUIRER Software Acquisition CMMModel Contract Performance Management (L3) Acquisition Risk Management (L3) Project Performance Management (L3) [PDSAP] Project Management (PM) L2 SAP SOL EVAL T2S Contract Track and Oversight Contract/SOW Scheduled SUPPLIER Deliverables Software CMMModel Software Product Peer Reviews RM Engineering (L3) (L3) SPP SPTO SCM SQA Organization Process Focus (L3) Integrated Software Management (L3) Intergroup Coordination (L3) and not recommend by SEI to take that path. Figure 3: Business Model. Organization Process Definition (L3) Training Program (L3) Training Program L3 Process Definition & Maintenance (L3) The Process Improvement Schedule With every new project initiative comes planning. SEI provides guidelines for planning process improvement based on information corroborated with the industry. The SEI routinely publishes statistical information ( based on data collected from the industry. SEI publications provide an indication that organizations typically take anywhere from months to move up a maturity level. This number was used to develop a schedule of activities to be performed for the AMO. The initial target of 18 months was selected based on the size of the AMO. This also provided a 6 month buffer for any unexpected turn of events, ensuring the project didn t prolong past the 2 year maximum. The activities were grouped into stages to establish milestones on the improvement schedule. Preparation activities followed the development schedule that included team training on the SA-CMM model, and team orientation on the stages of the process improvement schedule. The first stage of the improvement schedule was to conduct an initial assessment of the AMO against the SA-CMM model to obtain the introspective view. The initial analysis involved documentation reviews and interviews with the team members. Once all the information was consolidated, a gap analysis was performed on the results to identify both strengths to leverage and weaknesses to improve. An improvement plan and schedule was developed to correct the weaknesses identified. This was presented to the AMO team to understand the deficiencies that exist, and the work involved in correcting these deficiencies. The benefits were also outlined to gain buy-in from the team. Given the benefits were long term and processes were incrementally deployed, it was a challenge to demonstrate the benefits until these processes were institutionalised and practiced over a short period of time. Once the improvement schedule was executed, progress on the improvement activities was tracked against the schedule. Routine reporting to management was done that included accomplishments made, and issues impacting improvement activities for management resolution. As progress was made with improvements, assessment on the capability of the AMO provided an integrity check on the improvements made. With every CMM based process improvement, a series of informal assessments were planned for providing a continuous introspective view of the improvements. When the second informal assessment for the AMO showed compliance with all level 2 Key Process Areas, a final assessment was planned several months in advance to obtain a maturity rating for the AMO. The maximum effort required for this process improvement effort occurred during execution of the improvement activities. The time frame for this effort will vary for other organizations depending on their level of maturity. The amount of process documentation developed for the AMO was in the order of 100 documents, which required a substantial amount of effort to complete. Initial estimates for the improvement activities was set to 9 months, while actual was 12 months. Although the total estimated effort to reach maturity level 2 was 18 months, the actual effort (from the planning stage to the final assessment) was 24 months. This met the schedule requirements of the effort. 5. WHAT THE AMO GAINED FROM PROCESS IMPROVEMENT USING THE SA- CMM 5.1 Key Improvements Given the amount of effort that can potentially be required for process improvement, what can be gained from all this work? The first and foremost was achieving a structured and disciplined acquirer and supplier organizations; a matched team situation as shown in Figure 1. Both organizations are now better able to interactively perform their tasks. This is achieved in a number of ways. Communications has vastly improved between the AMO, the software supplier and with the user community which the AMO supports. This was noticeable by all the stakeholders and was brought up during progress reviews. The users have a much better insight and appreciation on the work involved for processing their needs. Progress reviews are scheduled on a regular basis, with information exchange between all the stakeholders. There is also an improved coordination of activities between the AMO and the software supplier. This is a key practice in maximizing software supplier s productivity and therefore a return on investment. The software supplier receives the proper guidance it needs to effectively perform the tasking needed. A master project schedule was established that identifies all the

5 tasks that have been scheduled for the software supplier. Monthly progress reporting of software supplier activities provides the AMO better oversight capability so that informed decisions can be made on any schedule deviations. One aspect of the monthly progress reporting that was noticeable over the 2-year effort was the movement and completion of tasks that were a backlog for several years. Both software and hardware upgrades on computer systems are an essential part of system maintenance in ensuring continued vendor product support, and availability of new product features. Because too much focus was placed on processing the high volume of user changes submitted, major maintenance upgrades fell behind. This also became a high-risk area because of system vulnerability, and the availability of vendor support slowly diminishing. Improved communication between stakeholders provided the understanding and the need for the upgrades, together with the guidance from the AMO to prioritise the software supplier s tasking in completing the upgrades. The master schedule now identifies future upgrades for continuity in system maintenance. Another noticeable improvement was the shift in software supplier tasking on the type of development work being done. In the past, the software supplier was tasked to spend most of its time on code modifications to existing production systems, while time for researching and developing new systems based on new technology was limited. The downside on this approach is that the technology of the existing systems falls behind. This inhibits the availability of new systems that can potentially provide a substantial number of system enhancements to the current systems. In some cases, new technology was found to provide solutions to user needs and problems that existed for AMO to effectively maintain the production systems. An example of such a problem is continuity of operations during disaster recovery. There were several unsuccessful attempts to restore the systems during routine trial runs, and had created a risk event. Because of the completion in the upgrades, disaster recovery trials are now successful. Establishing of process assets provided the AMO and the software supplier predictability in the acquisition and software development activities. As change requests (user requirements) were submitted to the AMO office, the team members in the AMO office and the software supplier now have a common understanding on the life cycle stages (which included the software supplier s development stages) of processing change requests, coordination of activities with all affected groups, and reporting on progress to management. A defined supplier s software development process and enhanced software testing process provided a more reliable production system. Software releases on production systems were monitored and controlled to ensure minimal impact on downtime. With these improvements in place, the reduction of production system downtime decreased from 5% in 1999 to less than 1% in This was significant as it supported the organization s mission and business objectives. Having documented processes (process assets) in place, and using automated tools to support the processes provided the mechanism for tracking and monitoring project activities and events. Examples of processes and tools include Risk Management plans and procedures for tracking risks during the acquisition, change request process to manage, track and monitor requirements through the change request life cycle process. The process assets provided the AMO office team members guidelines to follow when performing tasks that support the software acquisition. A configuration management tool was implemented to automate the change request life cycle process. All stakeholders have immediate access to submitted change requests and real time processing status. 5.2 Additional Functions and Capabilities Metrics reporting can play a key role in assessing and improving the effectiveness and utilization of processes. A quantitative management approach was taken by expanding the existing metrics reporting capability. Some data had been gathered in the past, but not effectively utilized. With the establishment of needed processes and documentation of existing processes, essential data collection for tracking and reporting key AMO and software development activities were identified. AMO defined reporting requirements from the software supplier for better assessment of software supplier activities, and progress in meeting scheduled contractual requirements. These reporting requirements were set in place, and monthly assessments are now made based on the progress reporting provided by the software supplier. There are some practices that may be considered as additional workload with no added value. Let s take one practice, for example, Quality Assurance (QA). QA is an overhead and sometimes perceived as not providing value to core business. There is a price tag involved with QA, which further elevates operating costs. Why would the AMO want to invest time and funding for QA? Understanding the value that QA can provide will demonstrate the benefits and return on investment for the AMO. As part of the Evaluation Key Process area of SA-CMM, products delivered by the software supplier to the AMO office must be evaluated against the requirements of the contract. To ensure the quality of products and services delivered by the software supplier meet the requirements of the contract, a QA function was established. The QA function provided guidance on the program to establish evaluation processes and practices, and ensure they are followed. AMO management have a much better oversight capability on the quality of delivered products with respect to contractual requirements. To maintain consistent and repeatable processes, a knowledge centre website was developed to house the

6 process assets in a controlled environment. As revisions are needed to bring process assets inline with current AMO practices, the updates are performed using the configuration management process. The knowledge centre website was readily accessible to all AMO team members, and used to institute repeatable practices. As staff turnover occurs, new team members will have access to the knowledge centre for relevant information. This will ensure continuity in the support that the AMO will maintain during the life of an acquisition. 6. SUMMARY The SA-CMM was created for the discipline of software acquisition, i.e., to help organizations improve their software acquisition processes. The SA-CMM focuses on how the acquisition organization manages its internal business, not on how the supplier does its business. The model was developed to be flexible enough in its application to be adaptive to a variety of organizations and their differing acquisition processes. Many of the principles documented in the model itself support this flexibility. In general, the SA-CMM has been successfully applied to most acquisition organizations and their unique processes. The AMO described in this paper illustrated one such organization that used the SA-CMM for its process improvement initiative. The model was used to obtain an introspective view of the AMO s ability to accomplish its acquisition mission, and developed a process improvement plan based on weaknesses identified from initial assessment findings and gap analysis. The plan was executed and weaknesses were corrected to fill in the gaps impacting the AMO s capability to manage acquisitions. Several informal assessments were conducted to determine resolution of weaknesses, and a successful final assessment was achieved with a SA-CMM level 2 maturity rating. There were several benefits gained from this process improvement effort. Significant benefits included the establishment of the Matched Team environment, where both the acquirer and supplier have a structured and disciplined environment interactively coordinating and performing their tasks. There is much better communication between all stakeholders through established processes, routine reporting and use of automated tools that capture data based on the change request life cycle process. The AMO provides the supplier the necessary guidance it needs to maximise the supplier s productivity, which provides a higher return on investment. This was achieved through supplier and acquirer coordination, planning and scheduling techniques. There were also some significant improvements in the way the AMO now does business. First there was the shift in supplier tasking in the type of development work performed. The supplier s tasking in maintaining the existing production systems based on current technology was being reduced, and more effort placed in identifying and developing systems based on new technology. Another improvement was documenting existing processes and establishing new ones needed to support AMO s capability for managing acquisitions. This provides the foundation for a common understanding by all AMO team members on tasking to be performed, coordination of activities with all affected groups, and reporting on progress to management. A knowledge centre website was also created to store these process assets in a controlled environment. The website was developed to provide accessibility to these process assets by all AMO team members. With all these improvements in place, there were noticeable improvements in the reliability and availability of production systems with a considerable reduction in system downtime. The results of the process improvement work done on the AMO demonstrated that when applied properly, the SA-CMM ensures that the acquisition organization is better poised to acquire the software product and services to meet the needs of the end users. REFERENCES 1. Cooper, J., et al., (1999) Software Acquisition Capability Maturity Model Version 1.03 (CMU/SEI-2002-TR-010 ESC-TR ). Pittsburgh, Penn.: Software Engineering Institute, Carnegie Mellon University. 2. CMMI Product Team (2001) Capability Maturity Model Integration (CMMI SM ) Version 1.1 (CMU/SEI-2002-TR-012 ESC-TR ). Pittsburgh, Penn.: Software Engineering Institute. Carnegie Mellon University. 3. Fisher, M., et al., (2002) Applying the SA-CMM, CrossTalk, Vol.15 No Paulk, M. et al., (1995) The Capability Maturity Model: Guidelines for Improving the Software Process. Addison-Wesley, Boston, Mass.