Software Quality 1
Software Quality What is Quality? Roger Pressman, a noted software engineering author and consultant, defines software quality as: Conformance to explicitly stated functional and performance requirements, explicitly documented development standards, and implicit characteristics that are expected of all professionally developed software The International Organization for Standardization (ISO) defines quality as The totality of features and characteristics of a product or service that bears on its ability to satisfy stated or implied needs. - (ISO 8402, 1986) 2 ER/CORP/CRS/SE01/003 Introduction What is quality? Roger Pressman, a noted software engineering author and consultant, defines software quality as: Conformance to explicitly stated functional and performance requirements, explicitly documented development standards, and implicit characteristics that are expected of all professionally developed software The International Organization for Standardization (ISO) defines quality as The totality of features and characteristics of a product or service that bears on its ability to satisfy stated or implied needs. - (ISO 8402, 1986) Note carefully that these definitions are not used to express a degree of excellence in a comparative sense; it merely identifies a product s quality with a subset of its attributes (which usually includes its price). The word 'implied' is also important: a decade ago, the emphasis was on products, and quality meant a product s ability to conform to specifications, i.e., stated needs. In the last several years, the word quality has been used increasingly to encompass features and characteristics that have a bearing on an organization s many stakeholders, including community, suppliers, shareholders, employees and management. Quality now includes measures of work life, workplace diversity, environmental conditions and competitiveness. A shorter definition is that a high-quality software system is one that s delivered to the users on time, costs no more than was projected, and, most importantly, works properly. Working properly implies that the software must be as nearly bug-free as possible. 2
What is Quality? Q: What is the Quality of a radio? 3 ER/CORP/CRS/SE01/003 What is the Quality of a radio? Throw this question to the house and wait for responses. You will get a lot of technical attributes like portability, clarity of sound, bandwidth etc. List them on the board. Typically, you will not get someone saying cost. So make a note of this and add cost to the list. The rationale is simply that cost is one of the attributes that have a bearing on the desirability! Point out that this set of attributes is a vector, so that the quality of a radio is nothing but a vector. So it is meaningless to talk of high quality or low quality just as it is meaningless to talk of a large vector or a small one. Each customer segment likes one particular kind of vector. A customer buying a portable radio for carrying it for picnic will want a vector with a large component of portability but a low component of cost and fidelity so that he can listen to cricket commentary while roaming. Finally, then, because quality is a vector, you can compare two vectors only if they are the same on some attributes but one is better on every other attribute. Bettering a product this way is what it means to improve quality 3
What is Quality? (Contd..) Please refer to Notes page for more explanation on previous slide 4 ER/CORP/CRS/SE01/003 It is worth mulling over what it means to "improve" on a product s quality. Is a pocket transistor radio costing Rs. 200 of lower quality than a high-fidelity monster costing Rs. 20,000? Or is it the other way round? To a large extent, the answer depends on how your existing and potential customers are defined, since it is they who mostly determine stated and implied needs. The ISO uses the term grade to indicate a planned difference in requirements, or, if not planned, a recognized difference. Thus, for example, the pocket radio and the hi-fi monster would belong to different grades since they are targeted at customers with distinct sets of requirements. We can, in general, only be sure that if each attribute of one radio is at least as good as the corresponding attribute in the other, then one of them can unambiguously be deemed the higher quality product. In turn, the process which makes the better product can be deemed to be the better process. 4
Need for Improvement Competitive Pressures Customer Satisfaction Management of Change 5 ER/CORP/CRS/SE01/003 Why improve on the quality of a product or process? Competitive Pressures: Today s business is extremely competitive, and the software industry is no exception. Companies must continuously make and sustain improvements in cost and quality to remain in business. Customer Satisfaction: Acquiring a new customer is far more expensive than retaining a current one. Further, few dissatisfied customers complain; the vast majorities simply take their business elsewhere. The Management of Change: In an industry like software where new tools and methods appear to arrive at a faster rate than it takes to train staff in their use, it is especially important that organizations fundamentally change their management styles and their workforce s attitudes so as to effect significant improvements. 5
Attributes of Quality To improve the Quality of software we first need to measure Quality To measure Quality we need to understand measure What and How Software Quality Factors are the What part of the measure. Software Quality Matrix are the How part of the measure 6 ER/CORP/CRS/SE01/003 Attributes of Software Quality In order to improve the quality of software first we should be able to measure the quality of software. To measure quality we need to understand the factors that affect quality. These factors should be sufficiently objective, rather than subjective and therefore measurable to help correctly evaluate the quality of the software. The collection of such measurable quality factors is called the Quality Matrix. Although there are many schools of thought with different set of factors we will discuss here McCall & Cavano s quality factors because McCall's work has been first steps towards development of Software Quality Metrics. We will also discuss quality Factors defined by ISO 9126 as they are the current standards. 6
Cost Of Quality (COQ) Why COQ? COQ is the quantification of the benefits of Improved Quality. In simple terms low COQ is higher benefits for same quality. What is COQ? Quality costs are the costs associated with preventing, finding and correcting defective work. Components of COQ Prevention Costs: These are costs of activities specifically designed to prevent poor quality Appraisal Costs: These are Costs of activities designed to find quality problems, such as code inspections and any type of testing Failure Costs: Costs that result from poor quality, such as the cost of fixing bugs, wasted efforts, cost of dealing with customer complaints, etc. 7 ER/CORP/CRS/SE01/003 Cost of Quality The motivation for improvement has been helped considerably by quantifying the benefits of improved quality into the language of money. This measurement is known as the cost of quality, and there are a number of ways in which it can be used: To quantify the costs of chronic waste; software that fails to be delivered falls in this category. It is estimated that fifteen percent of software that is written is never used. To quantify the benefits of using preventive techniques to avoid problems rather than using appraisal methods to correct them. It is often tempting to conclude, at the end of a cost-of-quality estimation exercise, that there is no point in improving quality because the associated costs are greater than the benefits from the improvement. This conclusion is often arrived at without properly estimating the future benefits from quality improvement. Further, cost of quality figures have an important limitation: they typically ignore the intangible but very real loss of goodwill from a customer. There are two well-accepted and documented ways of recording your quality costs which are described in the British Standard BS 6143: Guide to the economics of quality. These are the Process Analysis model and the Prevention, Appraisal, Failure (PAF) model. In the process analysis model, all the activities of the firm are first viewed as well-defined processes; the cost of quality is then deduced in terms of the cost of conformance or nonconformance to these processes, without making judgments about the efficiency or relevance of the processes themselves. In the PAF model, as the name suggests, costs are split into the three abovementioned categories. The following figure shows the behavior of these three cost components. As the figure shows, there is usually an optimum level of quality at any given point of time. 7
Cost Of Quality Contd. Total Cost of Quality Failure costs Appraisal costs Prevention costs c o s t Degree of quality 8 ER/CORP/CRS/SE01/003 Prevention Costs: These are costs of activities specifically designed to prevent poor quality (coding errors, design errors, bad documentation, etc). E.g. Staff Training, requirement analysis, accurate internal documentation, etc. These costs increase with increased degree of quality. 2. Appraisal Costs: These are Costs of activities designed to find quality problems, such as code inspections and any type of testing. E.g. Design Review, Code Inspection, White box testing, Black box testing, training testers, Beta testing, test automation, usability testing, etc. Appraisal costs are initially low because there is practically no appraisal - you just make and market come what may. When you realize the importance of quality, you institute testing practices and so incur some costs. So at that stage, increase in quality implies some increase in costs. When you improve quality even more, you are relying on prevention rather than appraisal, so that the further increase in quality is due to increased investment in prevention and reduced dependence on appraisal. 3. Failure Costs: Costs that result from poor quality, such as the cost of fixing bugs and cost of dealing with customer complaints. Internal failure costs are failure costs that arises before your company supplies the product to customer, e.g. bug fixing. External failure costs are failure costs that arises after your company supplies the product to customer, e.g. production problems, investigation of customer complaints, refunds, penalties and recalls, etc. Obviously these costs reduce with increase in degree of quality. 4. Total Cost of Quality = Prevention Costs+ Appraisal Costs + Internal failure Costs + External failure Costs. 8
International Quality Standards ISO: International Organization for Standardization CMM: Capability & Maturity Model CMMI: Capability & Maturity Model - Integrated TQM: Total Quality Management Six Sigma Malcolm Baldrige National Quality Award 9 ER/CORP/CRS/SE01/003 9
International Quality Standard: ISO ISO 9000 & ISO 14000 are Generic Management Systems Standards ISO Defines Quality System as: The organizational structure, responsibilities, procedures, processes and resources needed to implement quality management. -(ISO 8402, 1991) Benefits of ISO include Improved operations Improved margins External recognition Market expansion ISO 9000 is about Processes not Products. ISO 14000 is about Environmental Management. 10 ER/CORP/CRS/SE01/003 ISO ISO 9000 and ISO 14000 are Generic Management System Standards. Generic means that the same standards can be applied to any organization large or small, whatever its product and whether it is a private enterprise, a public administration or a government department. Benefits of ISO include: Improved operations Improved margins External recognition Market expansion To be able to understand ISO 9000 we need to know what a quality system is. ISO defines this as: The organizational structure, responsibilities, procedures, processes and resources needed to implement quality management. (ISO 8402, 1991) The ISO 9000 is a series of five international standards on quality management and assurance. The series consists of ISO 9000, ISO 9001, ISO 9002, ISO 9003 and ISO 9004. 10
The ISO 9000 Series ISO 9000, Quality management and quality assurance standards - Guidelines for selection and use is a road map for implementing ISO 9001 - ISO 9003. ISO 9001, Quality systems - Model for quality assurance in design/development, production, installation and servicing. ISO 9002, Quality systems - Model for quality assurance in production and installation. ISO 9003, Quality systems - Model for quality assurance in final inspection and test. ISO 9004, Quality management and quality system elements - Guidelines and other related documents describe recommended quality actions or guidelines. 11 ER/CORP/CRS/SE01/003 ISO 9000, Quality management and quality assurance standards - Guidelines for selection and use is a road map for implementing ISO 9001 - ISO 9003. ISO 9001, Quality systems - Model for quality assurance in design/development, production, installation and servicing is used by companies to certify their quality systems throughout their product-development cycle. ISO 9002, Quality systems - Model for quality assurance in production and installation may be used by a company whose products have already been marketed, tested, improved and approved. ISO 9003, Quality systems - Model for quality assurance in final inspection and test is for companies in which comprehensive quality systems may not be important or necessary, such as commodity suppliers. ISO 9004, Quality management and quality system elements - Guidelines and other related documents describe recommended quality actions or guidelines. The ISO 9001, 9002 & 9003 are contractual documents which describe compliance in shall language. The ISO 9004 uses should language. 11
The ISO 9000 Series (contd..) Please refer to Notes page for more explanation on previous slide 12 ER/CORP/CRS/SE01/003 It is important to know that ISO 9000 registration is for quality systems, not products. The assumption is that if quality systems exist, internal processes are controlled. This, however, does not warrant another assumption one may be tempted to make: that if these processes are controlled, they are capable and are improving, and that the products from these processes conform to customer requirements. In other words, having registered quality systems can be far removed from guaranteeing high product quality or satisfying customers. Thus, it is perfectly possible for a firm manufacturing concrete life jackets or paper raincoats to get its quality systems certified under ISO 9002, covering production and installation, although the products would not serve any perceived customer need. This, however, is not a major deficiency in the registration procedure: any processes in the quality system for design and development would have to include processes for recording and addressing customer feedback. If the targeted customers found the product not satisfying their requirements, their feedback would ultimately drive this firm to examine the suitability of making concrete life jackets or paper raincoats. 12
International Quality Standards: CMM What is CMM? It defines how software organizations mature or improve in their ability to develop software. CMM is software specific TQM model. Developed by Software Engineering Institute (SEI) of Carnegie Mellon University (CMU) in 1993 (CMM Ver. 1.1). Defines 5 levels of process maturity for a software organization Each level consists of a grouping of homogeneous Key Process Areas (KPAs) The relationship is such that the most benefit from a KPA can only be achieved when KPAs from lower levels are in place. The KPAs in each level enhance the effectiveness of the previous maturity level KPAs while strengthening the foundation for further process maturity growth. CMM Model enables an organization to reduce project risk and increase productivity and quality. 13 ER/CORP/CRS/SE01/003 CMM As ISO is generic management system standards CMM standards are specific to software or IT industry. CMM model is based on Total Quality Management or TQM. There are lots of definitions for CMM. One good one is this: It defines how software organizations mature or improve in their ability to develop software. The model was developed by Software Engineering Institute (SEI) of Carnegie Mellon University in late 80s. This model provides a structured view in a five-layer model of increasingly sophisticated practices for those working in software. Each level is focused on achieving certain objective and each level (except Level 1) has certain Key Process Areas or KPAs associated with it. Each level addresses levels of maturity exhibited by the project. The first version of CMM came in early 90s. Following level describes the various levels of CMM. 13
CMM Contd. Productivity and Quality Change Management Defect Prevention Product and Process Quality Engineering Process Project Management Heroes 2 Repeatable 1 Initial 4 Managed 3 Defined 5 Optimizing Standard, Consistent Disciplined Continuously Improving Predictable Project Risk 14 ER/CORP/CRS/SE01/003 The orange triangle in the diagrams is the Project Risk, as the organization achieves higher level of maturity the project risks reduces and productivity increases, the green triangle is productivity. The yellow boxes are the five levels of CMM. Level 1 Initial: There are no processes defined and success is person dependent. The process capability of this level, therefore, is defined as Heroes. Level 2 Repeatable: The focus area of this level is Project Management. Project standards are defined and followed. Projects functionality, scheduled and costs are tracked. Projects history data is available and used. Risks can be identified at early stages of the projects. Projects processes are therefore under control. The process capability of the organization at level two is disciplined. Level 3 Defined: The focus area of this level is Engineering Process. At this level the processes are defined and standardized at organization level. Projects tailor these standard processes to suit their need. The process capability of the organization at this level is standard & consistent. Level 4 Managed: The focus area of this level is Product & Process Quality. At this level quantitative quality goals are set for software products and processes at organization level. Organization wide Process Capability Data is maintained deriving data from completed projects. Projects use this data as a benchmark or quality goal. The process capability of the organization at this level is predictable. Level 5 Optimizing: The focus area of this level is Continuous Improvement. Organization has means to identify weakness and strengthen the process proactively, with a goal of preventing the occurrence of defect. The process capability achieved at this level is continuous improvement. 14
ISO vs. CMM Sr.No ISO CMM 1 ISO is a generic standard. It is applicable for all industries. CMM is very specific to software industry. 2 ISO is a standard. It tells what needs to be done, in an organization. CMM is a model. It doesn t mandate the practices 3 ISO focuses on the entire organization s processes CMM is specific to software processes 4 ISO Certification is followed by Surveillance audit once in 6 months After CMM assessment, there are no such checks. It is up to the organization to use it for internal process improvements 5 ISO is continuous CMM is staged (It has different levels of process maturity) 6 Internal Audits are mandatory Not mandatory 15 ER/CORP/CRS/SE01/003 ISO vs. CMM When applied to software industry what are the differences between the two internationally acclaimed quality standards, ISO and CMM? Though both ISO & CMM are based on the TQM model, both talk about Continuous Improvement (CMM stresses more on this), both stress on Say what you do. Do what you say, there are some obvious differences. 15
International Quality Standard: CMMI CMMI is CMM-Integrated. Multiple CMM models, like CMM for Software Development, CMM for Product Development, etc were integrated by SEI in 1998-99 to form a new model called CMMI. Structure of CMM & CMMI is same, both have 5 levels of maturity. Difference lies in the KPAs for each level. Level 1: Both CMM & CMMI has no KPA at this level, Level 2: CMM has 6 KPA while CMMI has 7 KPA Level 3: CMM has 7 KPA while CMMI has 13 KPA Level 4: Both CMM & CMMI has 2 KPA each. Level 5: CMM has 3 KPA while CMMI has 2 KPA In all CMM has 18 KPA while CMMI has 24 KPA. 16 ER/CORP/CRS/SE01/003 16
International Quality Standard: TQM Definition of TQM A management approach to an organization centered on quality, based on the participation of all its members and aiming at long-term success through customer satisfaction, and benefits to the members of the organization and to society. -(ISO 8402, 1991) Principles of TQM Quality is an organization wide process. Quality is what the customer says it is. Quality is an ethic. Quality is the most cost-effective, least capital-intensive route to productivity. 17 ER/CORP/CRS/SE01/003 Total Quality Management (TQM) Now as we know that ISO and CMM are based on TQM, let s see what TQM is. In the last ten years there has been a rapid acceleration in defining different approaches to the production of quality products that can all be categorized loosely as Total Quality Management (TQM). The term Total Quality Management was probably first adopted by the US Naval Air Systems Command in 1985, to describe the Japanese style management approach to continuous improvement. The ISO definition of TQM is A management approach to an organization centered on quality, based on the participation of all its members and aiming at long-term success through customer satisfaction, and benefits to the members of the organization and to society. (ISO 8402, 1991) Some of the canons that constitute this management approach are: Quality is an organization wide process. It is not a specialist function, a department, nor awareness or testing program alone. Quality is what the customer says it is. No one can compress in a market research statistic or defect rate the extent of buyer frustration or delight. Quality is an ethic. The pursuit of excellence, deep recognition that what you are doing is right, is the strongest human emotional motivator in an organization. Cold metrics alone are not enough. Quality is the most cost-effective, least capital-intensive route to productivity. The productivity concept, which was thus far confined to more, is enhanced to include the quality aspect of good. TQM, in being an approach, goes beyond mere quality management, which refers to the activities of the overall management function that determine the quality policy, objectives, and responsibilities, and implement them by various means. The software industry is still relatively young; as such, the application of TQM to software is still very much in its infancy. The industry is attempting to reach in 30 years a stage of maturity that has taken some industries over 200 years to achieve. This is illustrated in the following slide Ages of Quality. 17
Three Steps to TQM First Create a strategic objective that commits the entire company to total quality and customer satisfaction as the primary function of business. Second Determine what customers want. Third Design the products the way the customer wants to use them. Build quality into every product. Deliver on promises made to the customer. Ensure that customer orders are filled correctly and on time. Determine the correct price that gives the customers value. 18 ER/CORP/CRS/SE01/003 Although experts differ in their views on how best to implement TQM, there are a number of essential common elements of any TQM system. A prerequisite to the implementation of TQM is the establishment of effective planning, measurement and control mechanisms. The other common elements include continuous improvement, an objective definition of quality, commitment, teamwork, and, of course, training and education. The three steps to implement TQM are: FIRST Create a strategic objective that commits the entire company to total quality and customer satisfaction as the primary function of business. SECOND Determine what customers want. THIRD Design the products the way the customer wants to use them. Build quality into every product. Deliver on promises made to the customer. Ensure that customer orders are filled correctly and on time. Determine the correct price that gives the customers value. 18
International Quality Standards: Six Sigma The Six key concepts of Six Sigma are: 1. Critical to Quality: Attributes most important to the customer. 2. Defect: Failing to deliver what the customer wants. 3. Process Capability: What your process can deliver. 4. Variation: What the customer sees and feels 5. Stable Operation: Ensuring consistent, predictable processes to improve what the customer sees and feels. 6. Design: Designing to meet customer needs and process capability 19 ER/CORP/CRS/SE01/003 Six Sigma The word 'Sigma' is a statistical term that measures how far a given process deviates from perfection. The significance of Six Sigma is that if you can measure how many defects you have in a process, you can systematically figure out how to eliminate them and get as close to 'zero defects' as possible. The Six Key concepts of Six Sigma are: 1. Critical to Quality: Attributes most important to the customer Understanding Customer's needs and expectations by employing six approaches to communicating with customer Measure business performance against dynamic customer requirements and respond to changing market place conditions Quality function deployment (QFD) and failure modes and effects analysis (FMEA) can help identify critical to quality characteristics. 2. Defect: Failing to deliver what the customer wants Reducing the defect rate Determining the defect cost Mistake Proofing techniques eliminate the sources of errors and ensure that a process is free of defects. 3. Process Capability: What your process can deliver Understanding Process Capability principles and calculating Process Capability are integral to staying competitive and meeting customer requirements. 19
International Quality Standards: Six Sigma (Contd..) Please refer to Notes page for more explanation on previous slide 20 ER/CORP/CRS/SE01/003 4. Variation: What the customer sees and feels Two sources of variation : Common cause and special cause Multi-Vari Analysis offers a means of reducing large numbers of unrelated causes of variation to a family of related causes. Reducing common-cause variation so that the distribution has a very small standard deviation 5. Stable Operations: Ensuring consistent, predictable processes to improve what the customer sees and feels. Operational Excellence methodology for identifying the right projects, using the right people to lead projects, right tools and roadmap Turning quality into a management system Improving cycle time to process applications for long term disability benefits 6. Design: Designing to meet customer needs and process capability. Follow a four phase process to achieve design. Identify, Design, Optimize and Validate Design helps eliminate designed-in quality problems that account for 70-80% of defects Link Six Sigma with QS 9000. 20
Malcolm Baldrige National Quality Awards Baldrige Award Criteria Description Relative Weighting Customer focus on satisfaction Effectiveness of systems to determine customer requirements and demonstrated success in meeting them 30% Quality and operational results Results in quality achievement and quality improvement, demonstrated through quantitative measures 18% Human resource development and management Success of efforts to develop and realize the full potential of the work force for quality 15% Management of process quality leadership Effectiveness of systems and processes for assuring the quality of al operations Senior executives success in creating and sustaining a quality culture 14% 10% Information and analysis Effectiveness of information collection and analysis for quality improvement and planning 7% Strategic quality planning Effectiveness of integrating quality requirements into business plans 6% 21 ER/CORP/CRS/SE01/003 Malcolm Baldrige National Quality Award One response to the inroads Japanese and European competitors made in both U.S. and international markets during the 1980s was the creation of the Malcolm Baldrige National Quality Award. Widespread concerns that the quality levels of imported products exceeded American-made products led Congress to establish this prize in 1987 as part of a national quality improvement effort. Named after the late secretary of commerce, the Baldrige Award is the highest level of national recognition for quality that a U.S. company can receive. Its purposes are to promote quality awareness, to recognize quality achievements of U.S. companies, and to publicize successful quality strategies. The table in the slide lists the criteria for winning the Baldrige Award 21
Summary In this chapter we looked at: What is Software Quality What are the attributes and factors of Software Quality The Cost of Quality Parameters of Quality Matrix International Quality Standards like ISO CMM CMMI TQM Six Sigma & Malcolm Baldrige National Quality Award. 22 ER/CORP/CRS/SE01/003 22
References Pressman, R.S (1992)., Software Engineering: A practitioner s approach., McGraw- Hill, Singapore, 3 rd Edition. Gillies, AC & Smith, P (1994)., Managing Software Engineering: CASE studies and solutions., Chapman & Hall, London. PriDE CSQA Program @ Infosys 23 ER/CORP/CRS/SE01/003 23
Thank You! 24 ER/CORP/CRS/SE01/003 24