Software Quality Management Minsoo Ryu Hanyang University msryu@hanyang.ac.kr
Outline Software Quality Model Software Quality Management Process and Quality Quality Metrics 2 2
What is Quality? Quality, simplistically, means that a product should meet its specification This is problematical for software systems There is a tension between customer quality requirements (efficiency, reliability, etc.) and developer quality requirements (maintainability, reusability, etc.); Some quality requirements are difficult to specify in an unambiguous way; Software specifications are usually incomplete and often inconsistent 3 3
Software Quality (IEEE View) Software quality is: The degree to which a system, component, or process meets specified requirements The degree to which a system, component, or process meets customer or user needs or expectations [IEEE_Std_610.12-1990] 4 4
Software Quality (Pressman) Software quality is : Conformance to explicitly stated functional and performance requirements, explicitly documented development standards, and implicit characteristics that are expected of all professionally developed software [Pressman2003] 5 5
McCall s Quality Model (1977) McCall, Richards, and Walters studied the concept of software quality in terms of two key concepts as follows: quality factors and quality criteria A quality factor represents the behavioral characteristic of a system Examples: correctness, reliability, efficiency, testability, portability, A quality criterion is an attribute of a quality factor that is related to software development Example: Modularity is an attribute of the architecture of a software system A highly modular software allows designers to put cohesive components in one module, thereby increasing the maintainability of the system 6 6
McCall s Quality Model McCall et al. identified 11 quality factors 7 7
McCall s Quality Model The 11 quality factors have been grouped into three broad categories Product operation Product revision Product transition 8 8
McCall s Quality Model 23 quality criteria 9 9
McCall s Quality Model Relationship between quality factors and criteria Each quality factor is positively influenced by a set of quality criteria, and the same quality criterion impacts a number of quality factors Example: Simplicity impacts reliability, usability, and testability If an effort is made to improve one quality factor, another quality factor may be degraded Portable code may be less efficient Some quality factors positively impact others An effort to improve the correctness of a system will increase its reliability 10 10
McCall s Quality Model Quality criteria Traceability Completeness Quality factors Correctness Consistency Accuracy Reliability Error tolerance Execution efficiency Storage efficiency Access control Access audit Efficiency Integrity Operability Training Usability Communicativeness Simplicity Conciseness Maintainability Instrumentation Self descriptiveness Expandability Generality Modularity Software system independence Machine independence Communications commonality Testability Flexibility Portability Reusability Interoperability Data commonality 11 11
Factor-Criteria-Metrics-Model Factors (to specify) They describe the external view of the software, as viewed by the users Criteria (to build) They describe the internal view of the software, as seen by the developer Metrics (to control) They are defined and used to provide a scale and method for measurement 12 12
Quality Metrics 13 13
ISO Quality Model An international set of standards for quality management is called ISO 9000 ISO 9001 is an international quality management system standard applicable to organizations within all type of businesses ISO 9001 internally addresses an organization s processes and methods and externally at managing (controlling, assuring etc.) the quality of delivered products and services ISO 9001 is a process oriented approach towards quality management 14 14
ISO Quality Model 15 15
ISO Quality Model 16 16
Document what you do ISO 9000 Philosophy in conformance with the requirements of the applicable standard Do what you document Record what you did Prove it maintenance of registration requires audits every three years, with mini-audits every six months 17 17
Criticisms of ISO 9000 Many companies have found the transition to conforming to ISO 9000 difficult This, along with doubts about the fundamental value of the standard, has spawned many criticisms, including: The compliance process is costly and time-consuming Lots of administration is needed to implement it "When all you have is a hammer, every problem looks like a nail." It has been argued that it may not be appropriate to apply a process such as ISO 9000 to a field requiring creativity, such as software engineering, which is more analogous to designing factories than to operating a factory 18 18
Criticisms of ISO 9000 Many companies only register to ISO 9000 because they are forced to by the marketplace, whether or not ISO 9000 is in fact appropriate to their business. ISO 9001:2000 does not give too much practical advice but instead focuses on general principles. In order to create a standard applicable to almost any kind of organization, specific requirements and tools were avoided whenever possible. This is one of the reasons for the proliferation of industry specific standards which are more practical and give clear guidance about what quality tools have to be used when Toyota abandoned the standard in 2000, moving back to their in-house Toyota Production System 19 19
Defects and Software Quality Software errors are the cause of poor software quality [Galin2004] In software, higher quality (in the form of lower defect rates) and reduced development time go hand in hand [McConnell 1996] Unlike the low-defect kind of quality, attention to this kind of quality tends to lengthen the development schedule [McConnell 1996] Defects are negatively related to functionality and reliability Defects also interfere, to some degree, with other dimensions of quality 20 20
Defects and Software Quality cost to find and fix a defect 100 60.00-100.00 log scale 10 10.00 1 0.75 1.00 1.50 3.00 Req. Design test system code test field use 21 21
Outline Software Quality Model Software Quality Management Process and Quality Quality Metrics 22 22
Software Quality Management Quality Management System [ISO 9000]: The organizational structure, responsibilities, procedures, processes and resources for implementing quality management Quality management in three layers Software Quality Assurance Layer Establish organizational procedures and standards for quality Software Quality Planning Layer Select applicable procedures and standards for a particular project and modify these as required Software Quality Control Layer Ensure that procedures and standards are followed by the software development team 23 23
Three SQM layers by Ian Sommerville Software Quality Assurance (SQA) layer An organizational quality guide of Standards, regulations, and procedures to produce, verify, evaluate and confirm work products during the software development lifecycle Incorporated knowledge base of best practices Off-the-shelf software tools selected to apply the above Software Quality Plan (SQP) layer A project level quality plan written by each project for declaring project commitment to follow an applicable set of standards, regulations, procedures and tools during the development lifecycle In addition, SQP should contain quality goals to be achieved, expected risks and risk management Software Quality Control (SQC) layer Ensures in-process that both SQA and SQP are being followed by the development teams SQC activities include Mentoring how to produce artifacts, such as well-defined engineering documents using standard templates Mentoring how to conduct standard processes, such as quality reviews Perform in-process quality reviews to verify, evaluate and confirm artifacts Verify and evaluate to improve the use of methods, procedures and adopted software tools 24 24
Software Quality Management Quality management should be separate from project management to ensure independence 25 25
Quality Assurance Quality Assurance [ISO 9000]: All those planned and systematic actions necessary to provide adequate confidence that a product or service will satisfy requirements for quality Software quality assurance [IEEE]: A planned and systematic pattern of all actions necessary to provide adequate confidence that an item or product conforms to established technical requirements A set of activities designed to evaluate the process by which the products are developed or manufactured 26 26
Quality Planning A quality plan sets out the desired product qualities and how these are assessed and defines the most significant quality attributes The quality assurance plan should define the quality assessment process It should set out which organizational standards should be applied and, where necessary, define new standards to be used 27 27
Quality Control Quality Control [ISO 9000]: The operational techniques and activities that are used to fulfil requirements for quality Quality Control is the series of inspections, reviews and tests used throughout the development cycle to ensure that each work product meets the requirements placed upon it [Pressman2004] Commonly use the techniques we learned in V&V (inspection and testing) 28 28
Outline Software Quality Model Software Quality Management Process and Quality Quality Metrics 29 29
Process and Product Quality A fundamental assumption of quality management is that the quality of the development process directly affects the quality of delivered products In an automated manufacturing system, the process involves configuring, setting up and operating the machines involved in the process Once machines are operating correctly, product quality naturally follows 30 30
Process and Product Quality Software is not manufactured but is designed Software development is a creative rather than a mechanical process, so the influence of individual skills and experience is significant External factors, such as the novelty of an application or commercial pressure for an early product release, also affect quality irrespective of the process used However, experience has shown that process quality still has a significant influence on the quality of software Process quality management and improvement can certainly lead to fewer defects in delivered software 31 31
Process-based Quality 32 32
Six Sigma Six Sigma is a set of techniques and tools for process improvement It was introduced by engineer Bill Smith while working at Motorola in 1986 Jack Welch made it central to his business strategy at General Electric in 1995 The term six sigma is derived from six standard deviations 3.4 instances (defects) per million occurrences implying an extremely high quality standard The Six Sigma defines five core steps (DMAIC): Define customer requirements and deliverables and project goals via welldefined methods of customer communication Measure the existing process and its output to determine current quality performance (collect defect metrics) Analyze defect metrics and determine the vital few causes Improve the process by eliminating the root causes of defects Control the process to ensure that future work does not reintroduce the causes of defects 33 33
Strategy by Phase Measure Phase Process Characterization Measure (What) Step What is the frequency of Defects? Define the defect Define performance standards Validate measurement system Establish capability metric Focus Y Y Y Y Improvement Analyze Measure Improve Control Analyze (Where, When, Why) Process Optimization Improve (How) Where, when and why do Defects occur? Identify sources of variation Determine the critical process parameters How can we improve the process? Screen potential causes Discover relationships Establish operating tolerances Were the improvements effective? Re-establish capability metric X Vital X X Vital X Vital X Y, Vital X Analyze Analyze Measure Improve Measure Improve Measure Control Control Control (Sustain, Leverage) How can we maintain the improvements? Implement process control mechanisms Leverage project learning's Document & Proceduralize Y, Vital X Analyze Improve Control 34 34
SW Capability Maturity Model SW-CMM was developed by the Software Engineering Institute (SEI) of the Carnegie Mellon University SW-CMM defines a set of proven practices SW-CMM was initiated by the Department of Defense with the goal to obtain control of the quality of their software suppliers SW-CMM defines 5 maturity levels 35 35
SW Capability Maturity Model 36 36
SW Capability Maturity Model 37 37
SW Capability Maturity Model 38 38
ISO/IEC 15504 ISO/IEC 15504 (SPICE) A set of technical standards documents for the computer software development process and related business management functions Also known as Software Process Improvement and Capability determination (SPICE) ISO/IEC 15504 was initially derived from process lifecycle standard ISO/IEC 12207 and from maturity models like Bootstrap, Trillium and the Capability Maturity Model (CMM). ISO/IEC 15504 has been revised by: ISO/IEC 33001:2015 Information technology 39 39
The Assessment Framework Two-dimensional model for processes and process capability Process Dimension Process Categories Processes (P1,, Pn) Capability Dimension Capability Levels (CL1,, CL5) Process Capability Attributes Each process receives a capability level rating CL5 CL4 CL3 CL2 CL1 CL0 CUS.1 CUS.2...ORG.6 40 40
ISO/IEC 15504 Processes 41 41
ENG.1.4 Software Construction Purpose Produce executable software units and verify that they properly reflect the software design Outcomes Verification criteria will be defined for all software units against their requirements; Software units defined by the design will be produced; Consistency will be established between software requirements and design and software components; Verification of the software units against the design will be accomplished NOTE Part of this process is similar to the process Verification process (SUP.4) 42 42
The Measurement Framework Optimizing The process is continuously improved to meet relevant current and projected business goals Level 5 PA.5.1 PA.5.2 Optimizing Process Innovation Process Optimisation Predictable The process is enacted consistently within defined limits Level 4 PA.4.1 PA.4.2 Predictable Process Measurement Process Control Established A defined process is used based on a standard process. Level 3 PA.3.1 PA.3.2 Established Process Definition Process Deployment Level 1 PA.1.1 Level 2 PA.2.1 PA.2.2 Performed Process Performance Managed Performance Management Work Product Management Managed The process is managed and work products are established, controlled and maintained. Performed The process is implemented and achieves its process purpose Level 0 Incomplete Incomplete The process is not implemented or fails to achieve its purpose 43 43
Outline Software Quality Model Software Quality Management Process and Quality Quality Metrics 44 44
Quality Metrics You can t control what you can t measure Tom DeMarco, 1982 IEEE definition of software quality metrics: A quantitative measure of the degree to which an item possesses a given quality attribute A function whose inputs are software data and whose output is a single numerical value that can be interpreted as the degree to which the software possesses a given quality attribute Classification Process metrics: metrics related to the software development process Maintenance metrics: metrics related to software maintenance [Galin2004] Product metrics: metrics related to software artifacts 45 45
Process Metrics Software process quality metrics Error density metrics Error severity metrics Software process timetable metrics Software process error removal effectiveness metrics Software process productivity metrics 46 46
Error Density Metrics [Galin2004] 47 47
Error Severity Metrics [Galin2004] 48 48
Software Process Timetable Metrics [Galin2004] 49 49
Error Removal Effectiveness Metrics [Galin2004] 50 50
Process Productivity Metrics [Galin2004] 51 51
Maintenance Metrics [Galin2004] Help desk service (HD): software support by instructing customers regarding the method of application of the software and solution for customer implementation problems (depends to a great extent on user friendliness ) Related metrics: HD quality metrics: HD calls density metrics - measured by the number of calls HD calls severity metrics - the severity of the HD issues raised HD success metrics the level of success in responding to HD calls HD productivity metrics HD effectiveness metrics 52 52
HD Calls Density Metrics [Galin2004] 53 53
Severity of HD Calls Metrics and HD Success Metrics [Galin2004] 54 54
HD Productivity and Effectiveness Metrics [Galin2004] 55 55
Product Metrics [Galin2004] Corrective maintenance: Correction of software failures identified by customers/users or detected by the customer service team prior to their discovery be the customer (directly related to the software development quality) Related metrics: Corrective maintenance quality metrics Software system failures density metrics Software system failures severity metrics Failures of maintenance services metrics Software system availability metrics Corrective maintenance productivity metrics Corrective maintenance effectiveness metrics 56 56
Failures Density Metrics [Galin2004] 57 57
Failure Severity and Failures of Maintenance Services Metrics [Galin2004] 58 58
Availability Metrics [Galin2004] 59 59
Software Corrective Maintenance Productivity and Effectiveness Metrics 60 60
Product Metrics [Sommerville2004] Classes of product metric Dynamic metrics which are collected by measurements made of a program in execution Static metrics which are collected by measurements made of the system representations Dynamic metrics help assess efficiency and reliability; static metrics help assess complexity, understandability and maintainability 61 61
Static Metrics [Sommerville2004] 62 62
Software Product Metrics [Sommerville2004] Software metric Fan in/fan-out Length of code Cyclomatic complexity Length of identifiers Depth of conditional nesting Fog index Description Fan-in is a measure of the number of functions or methods that call some other function or method (say X). Fan-out is the number of functions that are called by function X. A high value for fan-in means that X i s tightly coupled to the rest of the design and changes to X will have extensive knock-on effects. A high value for fan-out suggests that the overall complexity of X m ay be high because of the complexity of the control logic needed to coordinate the called components. This is a measure of the size of a program. Generally, the larger the size of the code of a component, the more complex and error-prone that component is likely to be. Length of code has been shown to be one of the most reliable metrics for predicting errorproneness in components. This is a measure of the control complexity of a p rogram. This control complexity may be related to program understandability. I discuss how to compute cyclomatic complexity in Chapter 22. This is a measure of the average length of distinct identifiers in a p rogram. The longer the identifiers, the more likely they are to be m eaningful and hence the more understandable the program. This is a measure of the depth of nesting of if-statements in a program. Deeply nested if statements are hard to understand and are potentially error-prone. This is a measure of the average length of words and sentences in documents. The higher the value for the Fog index, the more difficult the document is to understand. 63 63
Object-Oriented Metrics [Sommerville2004] 64 64
Function Points A function point is a unit of measurement to express the amount of business functionality an information system (as a product) provides to a user Function points measure software size The cost (in dollars or hours) of a single unit is calculated from past projects Function points were defined in 1979 in Measuring Application Development Productivity by Allan Albrecht at IBM The functional user requirements of the software are identified and each one is categorized into one of five types: Internal Logical Files External Interface Files External Inputs External Outputs External Inquiries Once the function is identified and categorized into a type, it is then assessed for complexity and assigned a number of function points 65 65
Function Point Analysis Stage 1: Compute crude function points (CFP) Stage 2: Compute the value adjustment factor (VAF) for the project VAF varies between 0 and 70 Stage 3: Compute the number of function points (FP): FP = CFP x (0.65 + 0.01 x VAF) 66 66
CFP Calculation 67 67
VAF Calculation 68 68
Example: ATTEND MASTER 69 69
Example: ATTEND MASTER 70 70
Example: ATTEND MASTER VAF 71 71
Example: ATTEND MASTER FP = 81 x (0.65 + 0.01 x 41) = 85.86 72 72
Advantages & Disadvantages Main advantages Estimates can be prepared at the pre-project stage (over LOC) Based on requirement specification documents (not specific dependent on development tools or programming languages), the method s reliability is relatively high Main disadvantages FP results depend on the counting instruction manual Estimates based on detailed requirements specifications, which are not always available The entire process requires an experienced function point team and substantial resources The evaluations required result in subjective results Successful applications are related to data processing The method cannot yet be universally applied 73 73
FP and LOC FPs can be used to estimate LOC depending on the average number of LOC per FP for a given language LOC = AVC * number of function points AVC (average number of lines of code) AVC is a language-dependent factor varying from 200-300 for assemble language to 2-40 for a 4GL 3GL (third generation language): high-level language (C, C++) 4GL (fourth generation language): application-specific language (SQL) FPs are very subjective They depend on the estimator Automatic function-point counting is impossible Often appropriate for DB-based systems 74 74