Using the Systems Engineering Capability Maturity Model to Reduce Acquisition Risk

Transcription

1 Using the Systems Engineering Capability Maturity Model to Reduce Acquisition Risk Kerinia Cusick SECAT LLC Beach Blvd., #405 La Mirada, CA Abstract Determining who to award a contract to is a difficult process. A key issue is establishing the probability of the contractor delivering the product within cost, schedule and performance parameters. Various methods are used to determine the risk of the contractor not performing, but in the past most of these methods were a measure of the capability of individuals, not of the company s inherent ability to produce a product. As individuals move from one project to the next within a company the probability of success changes. The Systems Engineering Capability Maturity Model SM (SE-CMM) is a tool that procuring organizations can used to measure the capability of a company s systems engineering and management processes, thereby getting a more accurate representation of a company s ability to produce a product. There are two ways that the SE-CMM can be used effectively in the acquisition process, and both differ from how similar tools, such as the CMM for software, have been used in the past due to the different architecture of the SE-CMM. The first is to correlate a contractor s SE-CMM profile with the contract risk areas. The second is to use the organizational maturity information inherent in the SE-CMM to specify a desired profile that corresponds with the natural organizational improvement lifecycle. Both methods are discussed in this paper. Either method is an effective technique of getting a good understanding of a contractor s ability to perform to a contract, and using the SE-CMM will help reduce the risk associated the acquisition process. Introduction One of the key questions in awarding a contract is evaluating the capability the bidding contractor. What is the probability that they will be able to deliver the product on time, within budget, and meet all of the performance requirements? There are a number of tools that can be used to help answer this question. Past performance, an evaluation of the content of the contractor s proposal against specific criteria, personal relationships, and credibility of key management personnel are all tools that have been used to estimate risk during the acquisition process. The Systems Engineering Capability Maturity Model (SE-CMM) SM is a new tool that can be used to help reduce the risk of awarding a contract to someone that will not be able to deliver as promised. The problem with basing decisions strictly on past performance, proposal content, personal relationships, or credibility of specific individuals is that most of these are based primarily on the heroics of specific individuals. As people move and new teams are formed, the probability of successfully completing a project changes significantly. Organizations with strong processes that do not rely upon individual heroics to get a job done have a higher probability of performing SM Capability Maturity Model and CMM are service marks of Carnegie Mellon University

2 consistently from one contract to the next. Unfortunately there are a limited number of tools that measure processes. The success of the ISO-9000 series tools has been the ability to evaluate the strength of processes. The CMM for software has also been a strong tool to determine the maturity of software processes. The SE-CMM measures the strength of systems engineering and management processes. This paper does not propose that SE-CMM become a primary method for selecting a contractor, however it can be added as a helpful criteria for contractor selection. The SE-CMM was developed by a collaboration of systems engineering, modeling and assessment experts to provide industry with a tool designed to facilitate the improvement of systems engineering processes. The industrial collaboration, called EPIC, included GTE, Hughes, Loral, Lockheed, Software Engineering Institute (SEI), Software Productivity Consortium and Texas Instruments. Designed primarily for self-improvement the SE-CMM has a different architecture than the more commonly know SEI CMM for Software. Contrary to the single score system used by the CMM for Software, the SE-CMM used a rating profile with 18 individual scores. This has an impact on how the model can be used to help reduce risk. The SE-CMM measures the capability of 18 systems engineering and management processes (called process areas). Capability is a measure of repeatability, it is a measure of the probability of getting the same outcome if a process is performed repetitively. Taken as a whole, process capability is a measure of the probability of getting the same quality product from one project to the next. The higher the process capability score the higher the probability of getting the same quality, the lower the risk of working with the contractor. When an organization s systems engineering processes are evaluated against the SE-CMM, the result is complex. There is a profile of 18 scores, each with a value ranging from 0 to 5. How should these results be interpreted, how can the SE-CMM be best used to reduce acquisition risk? There are two primary concepts that should be considered in using the model for acquisition risk reduction: evaluating the specific risk areas of the contract being awarded, and the using the organizational maturity concepts inherent in the SE-CMM. Using Improvement Stages as a Risk Reduction Tool This method of using the SE-CMM to reduce acquisition risk is similar to how the CMM for software has been used. However, requiring an organization to be a level 3 makes sense when using the CMM for Software, the architecture of the SE-CMM precludes that from being a logical requirement. Since the SE-CMM was designed primarily for self improvement the author team was reluctant to use a single score system, and opted for a rating profile. The SE-CMM uses capability levels which are measured process area by process area, not maturity levels which imply performing specific process areas. While anyone not involved in model development finds this distinction oblique at best, it is an important point. The result is that requiring a contractor to perform all SE- CMM process areas at a capability level 2 or 3 does not really make sense. Procuring organizations need to recognize that some process areas in the SE-CMM are much harder to perform than others. Comparing process areas to classes, one would recognize that some of them are at the high school level while others are at the graduate or doctoral level. In a typical company, by the time some of the more difficult process areas (i.e. the graduate level classes) are being performed at a capability level 2 (person gets a B in the graduate class), the easier process areas (i.e. the high school level classes) will have most probably exceeded level 2 (as a graduate student would find it easy to get a A going back to high school).

3 Recognizing that some process areas are easier than others, one can use the content of the process areas and the concepts embedded in maturity levels to organize the process areas by levels of difficulty. Organizational maturity has stages. At the first stage an organization is simply doing the work that needs to be done. There is no planning, training, or defined processes, they just do what needs to be done. Work is performed in an ad-hoc manner, and success is based on individual heroics. The next stage is controlling local chaos, and instituting planning and tracking methods at the local, project level. The objective of this second stage is to ensure that organizational learning will actually occur. Projects capture what they are doing by documenting their processes so that they will be able to share what they learn with other projects. The third stage is to share learning across the organization and institute organization wide processes. The forth stage it to gather numerical data related to the processes being performed, and manage using that data. The fifth is to continuously improve using the data. The result of mapping all of the SE-CMM process areas to the concepts embedded in maturity levels is shown in Figure 1. To avoid confusion with the other terminology, SECAT personnel have started to refer to the structure shown in Figure 1 as Improvement Stages. Improve Org. Std SE Process Ensure Quality Provide Skills & Knowledge Manage SE Support Envmnt Manage Product Line Evolution Define Org. Std. SE Process Stage 3 Plan Tech Effort Monitor & Control Tech Effort Manage Risk Manage Configurations Integrate Disciplines Coordinate w/suppliers Stage 2 Stage 4 Stage 5 Verify & validate System Understand Ctsmr N & E Integrate System Evolve Sys. Architecture Derive & Allocate Rqmnts Analyze Candidate Solutions Stage 1 Capability Capability Capability Capability Capability Level 1 Level 2 Level 3 Level 4 Level 5 Figure 1-Improvement Stages- Mapping SE-CMM Process Areas to Maturity Levels Bringing some the process areas that focus on doing systems engineering (i.e. Analyze Candidate Solutions to Coordinate with Suppliers) to a capability level 1, is the easiest activity, which we call Improvement Stage 1. Improvement Stage 2 involves adding the project management process areas (i.e. Integrate Disciplines to Plan Technical Effort) and performing all of these process areas (Analyze Candidate Solutions to Plan Technical Effort) at a capability level 2. This is incrementally harder and the next logical step in overall improvement. The same concept of adding process areas at performing all of them at higher capability levels continues through Improvement Stage 5.

4 There is no intent to show an exact sequence for how a company would go about tackling improvement. Improvement Stages provide additional guidance, arranging the SE-CMM processes areas by order of difficulty. While some process areas map very nicely to one level, the concepts in some other process areas span multiple maturity levels. The Ensure Quality process area is a good example. Examining the content of the process area one can see that it includes examining the quality of the product and the quality of the process. Measuring the quality of the process is an organizational maturity level 4 concept, while quality of the product maps to level 2. To avoid forcing organizations to find methods of measuring process quality that do not make business sense, this process area was placed at Stage 4. If a procuring organization was comfortable with a company that used documented plans on each of its projects, but had no method of getting repeatability from project to project, a procuring organization would be interested in a company operating at Improvement Stage 2. If the procuring agency was interested in a company that had standards deployed across the organization for how they do systems engineering, the procuring agency would want Improvement Stage 3. This is not to say that all companies have to be operating at Improvement Stage 3. A risk analysis of the contract and contractor should be performed to determine what Improvement Stage is required for the contract. For example, if the project is unique for the contractor performing the project, there may not be that much value in having a standard deployed across the organization, therefore Improvement Stage 3. It is also important to note that Improvement Stage 3 represents a big jump from the current state of the practice. Very few companies are currently operating with organizational standard process for systems engineering deployed across the organization. Mapping Contract Risk Areas to an SE-CMM Profile A strength of the SE-CMM architecture is the ability to compare a contractor s profile against the predetermined contract risk areas. In this method of using the model, the procuring agency determines which are the high risk areas associated with the program. High risk areas are mapped to the SE-CMM process areas, and then evaluated against candidate contractor profiles. Companies that have strong processes, and a high SE-CMM score in the areas of high contract risk should be able to better perform the contract than a company with weak processes and low score. Looking at a couple of scenarios will clarify this concept. Upgrading an air traffic control system at a major airport carries its own unique set of risks that will be very different from other types of contracts. For example, technology is not typically a high risk item since typically commercial-off-the-shelf products are used, and no new technological barriers need to be broken to achieve product success. However, integration in this type of scenario is typically a high risk area. People, hardware and software need to be integrated to operate smoothly. Equipment ranging from regional radars, to operator workstations, to communication hardware need to be integrated with software, which all need to support the air traffic controller sitting at their station and the airplanes landing at the airport. If this is an upgrade to an existing airport integration is further complicated by the airport needing to remain functional during the integration process, with no danger posed to any flight traffic. This sort of system is also characterized by many customers with numerous needs. Customers will include the airport (and/or possibly the county or city), the air traffic controllers, the Federal Aviation Administration, and the airlines using the airport. Each of these group will have their own set of requirements for the system and the installation process. Some of these requirements

5 may directly or indirectly conflict. Understanding customer needs and working to resolve the conflicts is also a high risk area. In contrast, a technology demonstration contract for developing a vehicle designed to intercept incoming Intercontinental Ballistic Missiles in space, such as a kinetic kill vehicle, would have very different contract risk areas. Here the technology is a very high risk area since some sort of hardware and/or software break through is required to achieve success. Integration is typically not a high risk area since only a few vehicles are being built at most, each one a hand built unit. Certainly the issue of long term product support is not a question for this type of contract, but performing analysis and evaluating different solutions is critical to success. This is an example of how one can estimate the high risk areas based upon knowledge of the type of work that needs to be done. One can then map this to the SE-CMM process areas and come up with a table similar to the example shown in Table 1. Note that the example does not include all 18 SE-CMM process areas since the example becomes too complex. The two scenarios discussed above are included in Table 1. The air traffic control system upgrade is summarized as Air Traffic, and the ICBM kinetic kill vehicle is summarized as Interceptor. As can be seen in the table, each contract has very different risk areas based upon the nature of the work. Table 1- Risk Ranking for Both Contract Scenarios Risk Estimate Brief Rationale Process Area Air Traffic Interceptor Air Traffic Interceptor Analyze Candidate Solutions Low High Little new analysis required, significant design reuse Key task associated with R&D type contract Derive & Allocate Rqmnts Integrate Disciplines Integrate System Understand Csmtr Needs & Expctns Mntr & Cntrl Tech. Effort Plan Tech. Effort Manage Prod Line Evoltn Provide Ongoing Skills & Knowledge Coordinate with Suppliers High High Significant trades on which tasks allocated to operator, software, etc. Low High Job falls primarily on the integration and test team High Low Large scale complex system with complex integration schedule High Low Numerous customers with conflicting needs High High Replanning important to keeping airport operational throughout integration High Low Long duration integration effort, gradual integration of supplier provided subsystems Developing system that can meet the rqmnts is where most of the risk is Many disciplines involved in achieving technological breakthroughs Single unit, almost integrate as you go Single customer Significant replanning expected in research project High level planning accepted, with large level of effort specified High Low Long term support required No long term support required Low High Low degree of Requires people with specialization required knowledge of state of the art High Low Numerous subcontractors involved Few subcontractors involved

6 The table is intended only as an example for discussion purposes, and a fair number of simplistic assumptions have been incorporated to keep the example feasible. A more meaningful method of ranking should have some measure of resolution beyond the binary high/low risk system used in the table. The Delphi method is an example of a technique that could be used to develop this table. Once completed, this profile of high/low contract risk areas mapped to the SE-CMM process areas can be used to examine existing contractor SE-CMM assessment profiles. Figure 2 shows sample profiles of two possible suppliers interested in building a new air traffic control system or a kinetic kill vehicle. These two companies have already done assessments against the SE-CMM, and this data is available to the procuring organization. Coordinate w/suppliers Manage Prod Line Evoltn Plan Tech. Effort Mntr & Cntrl Tech. Effort Contractor 2 Contractor 1 Undstd Csmtr N&E Integrate Sys. Derive & Allocate Rqmnts SE-CMM Rating Figure 2- Two SE-CMM Assessment Profiles for Candidate Contractors The procuring organization evaluating proposals for the air traffic control system would evaluate the two contractor profiles against their high risk areas which were summarized in Table 1. The procuring organization could look and determine which contractor has more capable processes for each of their high risk areas, when complete having a table similar to the one shown in Table 2. A more detailed analysis and accurate ranking of risk would be helpful in evaluating the data shown in Table 2. Contractor #2 performs better in 5 of the 7 high risk areas. Contractor #1 has better processes in only one SE-CMM process area, Integrate System. If the vast majority of contract risk was only in system integration, the procuring organization may favor Contractor #1 even thought their score was consistently lower than Contractor #2. Clearly a tool like the SE- CMM does not replace the need for careful thought, but is does provide a method of quantifying risk versus capability.

7 Table 2- Contractor Comparison Air Traffic Control System High Risk Process Area Contractor with More Capable Processes Derive & Allocate Requirements #2 Integrate System #1 Understand Customer Needs & Expctns (same) Monitor & Control Technical Effort #2 Plan Technical Effort #2 Manage Product Line Evolution #2 Coordinate with Suppliers #2 Conclusion In the past the CMM for software has been referenced in Requests for Proposals requiring the organization to be a specific maturity level in order to submit the proposal. However, as shown in this paper, that is not appropriate for the SE-CMM. There are two primary ways that the SE- CMM can be used effectively and that is either to correlate a contractor profile to a specific contract risk profile or to specify an Improvement Stage. Either method is an effective means of assessing the capability of systems engineering processes in an organization and thereby reducing the acquisition risk.

8 Title: Author: Using the Systems Engineering CMM to Reduce Acquisition Risk Kerinia Cusick Director, SECAT LLC Beach Blvd., #405 La Mirada, CA phone/fax: Speaker: Kerinia Cusick