TRILLIUM V3.0 A MODEL FOR THE ASSESSMENT OF TELECOM SOFTWARE SYSTEM DEVELOPMENT CAPABILITY

Transcription

1 TRILLIUM V3.0 A MODEL FOR THE ASSESSMENT OF TELECOM SOFTWARE SYSTEM DEVELOPMENT CAPABILITY Alain A. April, François Coallier Acquisition Quality Engineering & Research 2265 Roland Therrien Longueuil, Québec Canada J4N 1C5, (T) (F) , (I) fcoallie@qc.bell.ca, aapril@qc.bell.ca To obtain a version of the Trillium v3.0 model : ABSTRACT : ftp://ftp.quality.org/pub/qc/trillium ftp://matrix.casti.com/pub/qc/trillium WWW: Since the early 70's Bell Canada has carried-out vendor assessments covering commercial and quality issues. The quality assessment has been traditionally oriented toward manufacturing and customer support functions. In the early 80's, with the growing importance of the software component in all products, the development part of the life-cycle saw its significance increased. The increasing complexity and dimension of telecommunication software s has led to numerous research activities and the development of quality tools in telecommunication product and services development. Since 1982 Bell Canada has invested toward developing a model to assess the software development process of existing and prospective suppliers as a mean to minimize the risks involved and ensure both the performance and timely delivery of software system purchased. Bell Canada telecommunication network operations depends on more than 50 million lines of code. This number increases yearly as the network becomes more digital and the operational support systems are introduced to mechanize the administration. The median size of these software systems lies between 200 To 400 Kloc (thousands of lines of code), with the largest having over 20 million lines of code. Most of this software is real-time, with a majority of the systems of a hybrid nature (combination of real-time, database and graphical). In this environment some software s control safety/critical systems and must be reliable in order for Bell Canada to offer a level of service required by its customers. In April 1991 the first version of the TRILLIUM model was elaborated by Bell Canada, Northern Telecom and Research Bell Northern. This model and its application method have been improved based on experience and feedback from suppliers. From 1992 to 1993 versions 2 and 2.3d were used extensively and data about assessment was collected from assessment teams and management. In January 1995 version 3.0 was issued to Bell Canada suppliers and was placed in the public domain. This paper presents a summary of improvements included in version 3.0. It also presents the current research activities being carried in the Quality Engineering & Research group of Bell Canada associated with software process assessments. An assessment method is in itself a product designed from specifications. The specification criteria should be driven by the expectations that the assessing organization has from the supplier organizations that will be assessed. In software development it is essential that practices & methods used be measurable according to a set of well defined standards that are used industry wide.

2 Having a good development process (in the classical sense) will contribute very significantly to an increase in the capability of a software product development organization. But other factors are important and must be considered : Personnel, Technology, Management, development environment & tools. Traditional assessment approaches were typically process oriented a series of actions or operations conducting to an end. The assessment would focus on actions and operations and how they contribute to a desired purpose. When adding environmental factors in the assessment method we align with a "TQM" definition of process which is taking in account the mechanisms and resources as a whole. Technology has a significant impact in a discipline as young as software engineering. The state of the art in software engineering is far behind compared to other areas of engineering such as electronic design and development. For some authors it is so far behind that they declare that there is presently no software engineering discipline per se. The basis of the process assessment models for software rely on benchmarking. Comparing your practices with the best and successful organizations. It is also a basic tool that you will find in the TQM literature. For software assessment we have used initially two level of benchmarks to develop the model: Professional, national and international standards; Comparisons to other organizations in the same market segment. The software assessment model should therefore map to existing engineering standards as well as quality standards. It should also provide an output that can be used easily to benchmark against "best-in-class" organizations. 1. MODEL PURPOSE TRILLIUM can be used as a customer focused benchmark for either: -auditing the product development and support capability of a supplier of telecommunication -an internal product development and support capability continuous improvement program for organizations developing and supporting telecommunication products. The Trillium model has been developed from a customer perspective, as perceived in a competitive, commercial environment. In this context, Capability refers to: The ability of a development organization to consistently deliver a product or an enhancement to an existing product: that meets customer expectations, with minimal defects 1, for the lowest life-cycle cost, and in the shortest calendar time. Product in the TRILLIUM context refers to what the customer receive, use and perceive. For an embedded telecommunication product, this would include: hardware, software, documentation, training and support services. The ultimate objective of improvement programs initiated as a result of a TRILLIUM assessment is increased customer (and shareholder) satisfaction, rather than rigid conformance to the standards referenced by this document. A higher capability, in the TRILLIUM context, means for customer organizations that: a) the development organization is more responsive to customer and market demands, b) the life-cycle cost of the product(s) is minimized, and c) end-user satisfaction is maximized. For the development organization, achieving a higher capability can result in: a) lower development and maintenance costs, b) shorter cycle time and development intervals, 1 In telecommunications and other industries defects resulting in system failure or service outages are unacceptable. However, certain defects may be tolerable provided they have no impact on service levels and operations.

3 c) an increased ability to achieve content and schedule commitments due to effective project risk analysis and effort estimation, and, d) an increasing ability to meet quantifiable design and quality objectives at all stages of the development process. 2. MODEL SCOPE The TRILLIUM model is a set of practices derived from a benchmarking exercise. The scope of these benchmark aim at all practices that would contribute to an organization product development and support capability as defined previously. The principal sources of benchmarking information are, in decreasing order of importance: 1. version 1.1 of the Software Engineering Institute's Capability Maturity Model; 2. ISO 9001:1994 International Standard and : 1991 Guideline; 3. Bellcore's TR-NWT : Issue 2, june 1993 and TA-NWT : Issue 1, Decembre 1993; 4. Malcom Baldrige National Quality Award, 1995 criteriaè 5. IEEE Software Engineering standards: 1993 Edition and IEC 300: To understand the Trillium model, it is desirable to have a background in product engineering and quality management, and a solid understanding of the source documents listed above. The TRILLIUM Model covers all aspects of the software development life-cycle, most system and product development and support activities, and a significant number of related marketing activities. TRILLIUM has been designed initially to be applied to embedded software systems such as telecommunications systems. Although TRILLIUM is aimed at embedded software systems, much of the model can be applied to other segments of the software industry such as Management Information Systems (MIS). A significant percentage of the practices described in the model can be applied directly to hardware development. 3. MODEL FOUNDATION The TRILLIUM Model is based on the Software Engineering Institute (SEI) Capability Maturity Model (CMM) version 1.1. In the TRILLIUM context, the product development and support processes are viewed as an integral part of the organization's business processes. The architecture of the TRILLIUM Model (described in Section 3) differs significantly from the one of the CMM version 1.1. The most significant differences are: a roadmap based concept [instead of key product areas], a product perspective [instead of software], a wider coverage of capability impacting issues, a strong customer focus. The TRILLIUM Model incorporates additional practices from the following topics: Quality Management Business Process Engineering Technological Maturity Development Environment Systems Engineering Co-Engineering Concurrent Engineering Reliability Engineering Customer Support/Partnership Usability Engineering 4. TRILLIUM ARCHITECTURE The TRILLIUM Model is based on the concept of a roadmap. A roadmap is a set of related practices that focus on an organizational area or need, or a specific element within the product development process. Eache roadmap represents a significant capability for a software development organization. Within a given roadmap, the level of the practices is based on their respective degree of maturity. The most fundamental practices are at a lower level whereas the most advanced ones are located at the higher level. An organization will mature through the roadmap levels. Lower level practices must be implemented and sustained for higher level practices to achieve maximum effectiveness.

4 Roadmaps are organized into Capability Areas. Each represents a significant capability for a software development organization. There are 8 Capability Areas (see model for details) that can span the 5 TRILLIUM Capability levels. A representation of the span of each Capability Area is shown in table 3. Each of the 5 levels can be characterized, in a commercial environment, as follows: Lvl Level Name Risk Interpretation Unstructured High ad-hoc development process 2 Repeatable and Project Oriented Medium basic project based process 3 Defined and Process Oriented Low state of the art development process 4 Managed and Integrated generally difficult to achieve now 5 Fully Integrated technologically challenging Figure 1: TRILLIUM Capability Levels. LEVEL 1 LEVEL 2 LEVEL 3 LEVEL Process Ad-hoc Project based Organization-wide Standards None IEEE(), SEI L2+ IEEE(), SEI L3+ SEI L4+, Bellcore ISO 9001 IEC 300 (software)(c) TR-NWT (75%) IEC 300 (system) TR-NWT Process Improv. None Unstructured Deployed Systematic Table 2: TRILLIUM Level/Key Industry Indicators. For the purpose of TRILLIUM, note that the IEEE Software Engineering Standards, which mostly work products (e.g. Software design description, project management plan, etc.) oriented, are to be used as guidelines only. Trillium is based on Capability Areas, Roadmaps and Practices(see table 3). Each Capability Area incorporates one or more Roadmaps. Each Roadmap comprises one or more Practices that span several TRILLIUM levels. The complete version 3.0 model has 8 Capability Areas, 28 Roadmaps and 371 Practices. Capability Area Roadmap Level Span Organizational Processes Quality Quality Management 2-4 Business Process Engineering 2-3 Human Resource Development Human Resource Development and Management and Management 2-4 Process Process Definition 2-4 Technology Management 2-4 Process Improvement & Engineering 2-5 Measurements 2-4 Management Project Management 2-4 Subcontractor Management 2-3 Customer-Supplier Relationship 2-3 Requirements Management 2-4

5 Estimation 2-3 Quality System Quality System 2-5 Development Practices Development Process 2-5 Development Techniques 2-5 Internal Documentation 2-4 Verification & Validation 2-4 Configuration Management 2-5 Re-Use 2-5 Reliability Management 2-4 Development Environment Development Environment 2-5 Customer Support Problem Response System 2-3 Usability Engineering 2-4 Life-Cycle Cost Modeling 2-3 User Documentation 2-3 Customer Engineering 2-3 User Training 2-3 Table 3: Capability Area/Roadmap Relationship experts and validated in a peer review 6. BENCHMARK INTEGRATION process. The TRILLIUM model was built by integrating practices as per the following algorithm: 1. Practices are taken from the SEI CMM Version A mapping is performed between the CMM practices and ISO 9001 and ISO clauses. CMM practices may be modified to accommodate mapping. 3. From ISO 9001 and ISO clauses that cannot be mapped to CMM practices are extracted, added and integrated. 4. Bellcore standards clauses are mapped to the practices generated by steps 1, 2 & 3. Some practices may be adjusted to accommodate mapping. 5. From Bellcore standards clauses that cannot be mapped practices are extracted, added and integrated. 6. The same process is repeated with relevant portions of the Malcom Baldrige examination criteria. 7. Practices from IEC 300 are added. 8. Pointers to relevant IEEE standards are added. 9. Trillium specific practices are added to provide coverage of additionnal areas important to the telecommunications industry. These are based on professionnal benchmarks generated through the consensus of subject matter When practices are extracted from the CMM, they go through the following transformation if applicable: 1. The practice is generalized by either removing references to "software", or replacing them by "product and services" or "systems". 2. The practice is generalized by either removing references to "development", or replacing them by "development and support". 3. References to "group" or other specific organizational units are replaced by "function". 4. Allusions to specific documents are replaced by allusion to a process (e.g., "quality plan" by "quality planning") or to "documentation" or "information". The same type of transformations were applied when extracting practices from other standards. Assignation to a given level is based on the general guidelines of Table 1. Practices that are considered fundamental for the successful conclusion of a development project are assigned to Level 2.

6 Practices that are considered to be organization-wide in scope or fundamental to the continuous improvement of the development process are assigned to a level 3. Practices that deal wit CASE technology or characterize advanced process maturity (e.g. change management, integration of defect prevention, statistical process control and advanced metrics) are generally assigned to level 4. Level 5 practices typically deals with advancing technology as it applies to process automation, fornal methodologies and srategic utilization of organization repositories. 8. APPLICATION PRINCIPLES The TRILLIUM model should be always used in a pragmatic fashion keeping in mind that the bottom-line in all TRILLIUM based activities or program is improving customers (and shareholders) satisfaction. All TRILLIUM based activities should thus be context driven. This means taking into account factors such as: the nature of the product, the context in which the product will be used, the perception that the customer(s) has of the product (or products line) and its evolution, the pertinent development and support organizations. This imply that applicability of some practices and their implementation strategy will changes in function of the above factors. The professionalism of the staff and the organization involved in TRILLIUM based activities is thus critical. All TRILLIUM based activities should be leaded by experienced engineering personnel having TickIt lead auditors, practical knowledge of ISO auditing guidelines and SEI CMM self-assessment formal qualifications or equivalent. Malcom Baldridge assessor and QAI CQA, or equivalent, would also be good assets. Customers perceive a supplier organization as a black box delivering products and services (e.g., Marketing, Sales, Customer Support, Research & Development [hardware, software, system, silicon], Quality Assurance, etc.). TRILLIUM based activities should thus, ultimately, make abstraction of internal organizational boundaries in their application. In a TRILLIUM context, internal organizations such as marketing, engineering, customer support, etc... are all contributing to customer (and shareholder) satisfaction. The customer will always perceive more acutely the impact of the weakest contributor. All those organizations should thus, for instances adhere to a systematic improvement program with common and compatible goals. 9. TRILLIUM ACTIVITIES The following TRILLIUM based activities can be performed: supplier capability evaluation, participation in supplier capability joint-assessment, participation in supplier capability selfassessment participation in supplier capability continuous improvement program. While the first activity does not require that a special relationship exist between a customer and its supplier, the next three requires and increasing level of partnering to be really effective. A succinct description of these activities follows. Readers requiring further information should consult the references and take the appropriate training. a) Supplier Capability Evaluation The objective of this activity is to assess, by way of an auditing exercise using the TRILLIUM model as the reference benchmark, the product development and support capability of a supplier. This activity is performed as per recognized auditing practices such as ISO , but as

7 per the application principles described in section 2. TRILLIUM supplier capability evaluation are performed for: assessing the risks associated with the procurement of a given product, and incidentally doing business with its supplier, monitoring a quality/capability improvement program as part of its acquisition supplier quality management program. To be successful, such an evaluation need: o a assessor or team of assessor with adequate qualifications, experience and skills, o a minimum of cooperation from the supplier, o adequate preparation. b) Supplier Capability Joint-Assessment A joint-assessment in the TRILLIUM context is an assessment with the difference that it is performed by both customer and suppliers as a team exercise. This implies that the conclusions and recommendations of a TRILLIUM jointassessment represent a consensus of the joint team. To be successful, a joint-assessment need: team of assessor with adequate qualifications, experience and skills, adequate partnering maturity between both organization, clear and unequivocal buy-in from senior management in both organizations. A TRILLIUM joint-assessment that meet the above criteria will be far more efficient and reliable in getting a comprehensive appraisal of the supplier organization, but will require more resources that the normal assessment. With skilled personnel, a joint-assessment will be very efficient in appraising a low maturity organization (20/80 or Pareto rule). c) Supplier Capability Self-Assessment A TRILLIUM capability assessment is essentially an SEI self-assessment (as described in SEI-89-TR-7) with the following differences: at least one customer representative is on the self-assessment team. This representative is adequately qualified and has an improvement focused attitude, customer issues are systematically considered by the self-assessment team thought all phases of the self-assessment the self-assessment is implemented as per the application principles described in section 2, the TRILLIUM model is used as the reference benchmark. Like a SEI self-assessment, a TRILLIUM capability assessment is very demanding resource-wise, but is an efficient initiator of a capability improvement program because of the buy-in it is generating in the organization for its findings and recommendations. d) Supplier Continuous Improvement Program A capability continuous improvement (CI) program is essentially a Deming PDCA cycle.a TRILLIUM capability continuous improvement program is a Deming PDCA cycle were: at least one customer representative is on the CI program steering and management teams. This representative is adequately qualified and has an improvement focused attitude, customer issues are systematically considered by the CI teams thought all phases of the CI cycle. the TRILLIUM model is used as the reference benchmark, the "Check" part of the cycle is always associated with the contribution to customer (and shareholder) satisfaction. In order to achieve the highest possible capability, development and support organizations must strive to:

8 a. make the continuous improvement philosophy part of the organization culture, and ensure that the proper mechanisms and processes are in place to support and encourage this culture, b. engineer and optimize their processes to meet customer or market requirements and objectives for the specific product and/or service, c. continuously optimize the resources available for each development project. The presence of an efficient capability continuous improvement program in a supplier is, for Bell Canada, a reassuring sign of organizational maturity. Being invited to participate in such a program is a sign, from a supplier quality management perspective, of high maturity in the customer-supplier partnership. The efficiency of a capability CI program should be measured in term of: contributions to customer (and shareholder) satisfaction improvement, improvements in capability maturity. 11. EXPERIENCE WITH THE MODEL We have asked an organization that has used the Trillium model, for the first time, to give us a shot description of their experience. Bell Canada has reorganized its operations to pursue outsourcing opportunities in both telecom and data processing. Under the restructuring, Bell Sygma Inc. will be the financial holding and management company for Bell Sygma Telecom Solutions, Bell Sygma Information Services, and Bell Sygma Systems Management. This first Trillium joint assessment was conducted on the billing organization of Bell Sygma Telcom Solutions during in Bell Sygma s official Quality Policy consists of a set of principles defined to help maintain its competitive edge at the forefront of the industry of software development and outsourcing. In 1994 the organization has decided to implement the Trillium model as a key component of a continuous improvement program. The initial step consisted of using Trillium in a Capability Joint-assessment exercise. Training for the exercise included a TickIT lead auditor course. Trillium is gaining wide acceptance in the organization by both developers as well as process support personnel, because of the applicability of the model on the current development infrastructure. It is a ready made framework that allows easy focus on specific issues within well defined contexts. As a result, the joint assessment team did not need to spend large amount of effort in preparing material to support the exercise, and it was easy to communicate with developers. The participation of the developers was improved because they recognized that the joint assessment team was using formal, comprehensive and well designed model and method. The joint assessment exercise has convinced Bell Sygma of the value and viability of using the Trillium model. Bell Sygma plans on integrating Trillium to the corporation s continuous improvement process. A Trillium (Camelia) joint-assessment is also planned with Bell Sygma Systems Management in fall FUTURE VERSIONS The next experimental release of TRILLIUM will be version 3.1d. This version will essentially be 3.0 with added capability areas to address the MIS, Data Center Operations, Internal Communications Networks (LAN/WAN) practices. Version 3.1d will be published in a Bell Canada edition. A Québec-France cooperative project, titled CAMELIA, funded by both the Québec and French government is aiming at optimizing TRILLIUM to the Management Information System focus. CAMELIA v1.0 will be trialed in summer 1995 with a full model, training and experiments in Québec and France. The following topics are covered in Camelia in addition to the coverage of Trillium 2.1 : Re-Engineering. Business Process Engineering Systems Architecture Data Management, Data Center Management,

9 Software Maintenance ACKNOWLEDGMENTS TRILLIUM is the result of the cooperative work and contribution of many individuals. The authors would like to specially thank Armelle LeGuevel, Denis Bistodeau, Richard Basque and Michel Kwas for their work on Camelia. Neil Gammage, Allan Graydon, Jean-Normand Drouin, Jean Mayrand, John WIlson, Joe Hatz and Richard McKenzie for their work on Trillium v3.0. The authors would like to thank Aubrey Yearwood for his description of the Bell Sygma s experience with the Trillium model.