B.H. Far

Transcription

1 SENG 521 Software Reliability & Software Quality Chapter 4: Software Quality Department of Electrical & Computer Engineering, University of Calgary B.H. Far

2 Contents Software quality Software quality models: Boehm s model, McCall s model, ISO 9126 model, etc. Software quality standards and metrics Measuring customer satisfaction Software quality assurance (SQA) Software quality management far@ucalgary.ca 2

3 What is Quality? Quality popular views: Something good but not quantifiable Something luxury and classy Quality professional views: Conformance to requirement (Crosby, 1979) Fitness for use (Juran, 1970) 3

4 Software Quality: Difficulties Need to account for creativity in the design of the product and the requirements rather than the product itself. Leonardo da Vinci Mona Lisa (1479) Kind of art Which one has a higher Pablo Picasso quality? Dorra Maar (1937) far@ucalgary.ca 4

5 What is Software Quality? /1 What SW does What is Specified What users need Conformance to requirement The requirements are clearly stated and the product must conform to it Any deviation from the requirements is regarded as a defect A good quality product contains fewer defects Fitness for use Fit to user expectations: meet user s needs A good quality product provides better user satisfaction Both Dependable computing system far@ucalgary.ca 5

6 What is Software Quality? /2 Various viewpoints/perspectives: i i Conformance to customers requirements Requirement, design and development quality Process quality vs. end-product quality Process quality: higher usability and dependability End-product quality: less failure Internal vs. external characteristics Rl Relativity: ii advantage over similar il products Conformance: conformance to standards far@ucalgary.ca 6

7 Definition: Software Quality ISO 8402 definition of quality: The totality of ffeatures and characteristics of a product or a service that bear on its ability to satisfy stated or implied needs ISO 9126 definition of quality: The totality t of features and characteristics ti of a software product that bear on its ability to satisfy stated t or implied needs far@ucalgary.ca 7

8 Software Quality: Classification By observation: Internal quality (quality while the product is bi being produced, dincluding ldi process and checks) k) External quality (final product quality) By process: Design quality Implementation quality Test quality Maintenance quality 8

9 Software Quality Models

10 Modeling Software Quality Purpose: To make the concept quality operational via refinement and measurement Selecting some attributes (or factors) Plotting relationship among attributes (many-to- many relationships) or Selecting some criteria (or intermediate and primitive constructs) to represent the attributes Mapping criteria (or primitive constructs) to metrics 10

11 Quality Model Structure User oriented SW Quality Quality Factor 1 Quality Factor 2... Quality Factor n Software oriented Quality Criterion 1 Quality Criterion 2 Quality Criterion 3... Quality Criterion m Measures Kapitel far@ucalgary.ca

12 Example: Attribute Expansion Design by measurable Quality objective objectives: Incremental design is evaluated to check whether the goal for each increment was achieved. Availability User friendliness % of planned Days on job to System uptime learn task supplied By new system Worst: 95% Best: 99% Worst: Best: 7 days 1 day far@ucalgary.ca 12

13 Example: CUPRIMDA Model Quality parameters (parameters for fitness): Capability Usability Performance Reliability Installability Maintainability Documentation Availability Reference: S.H. Kan (1995) 13

14 Example: Boehm s Model Figure from Fenton s Book far@ucalgary.ca 14

15 Example: McCall s Model (11) (25) (41) Figure from Fenton s Book far@ucalgary.ca 15

16 McCall Model Elements McCall quality model is organized around three types of Elements: Quality Factors (To specify): They describe the external view of the software, as viewed by the users. s. Quality Criteria (To build): They describe the internal view of the software, as seen by the developer. Quality Metrics (To control): They are defined and used to provide a scale and method for measurement. far@ucalgary.ca

17 SATC* Software Quality Model *Software Assurance Technology Center (1996) NASA Goddard Space Flight Center 17

18 Quality Model: ISO

19 Flexible Quality Model No need dto stick kto a fixed d( (published) model. You can define your own quality model based on users (customers and company) consensus. The model will be composed of quality attributes which are important for the given product. The model should be verified by actual measurement. far@ucalgary.ca 19

20 Process-oriented oriented Quality Models Purpose: To characterize process quality Measure process and product attributes Examples of Process Attributes: Effort (person-months) o s) Duration (months) Examples of Product Attributes: Code size (#classes) Test set size (#test cases) Correctness (#defects) far@ucalgary.ca

21 0 Process Defects # Example: Defect Distributions Combination i of (i)defect number, (ii) injection to detection period, and (iii) defect origin can be used to measure development (process) quality. Goal: Reduce number of defects that cross process step boundaries activity of defect discovery # Coding defects # Design defects # Requirements spec. defects far@ucalgary.ca

22 Effort Distribution # Pers son-hours s Development effort distribution over process steps should make rework shares explicit. Goal: Detect and reduce rework. Design Coding Test Process Activity Test triggered rework on Design and Code Coding triggered rework on Design Normal effort of activity far@ucalgary.ca

23 Software Quality Standards

24 Software Quality Standards Software quality management standards (e.g., ISO 9000 series) covered in SENG 511 Software process quality assessment standards (e.g., ISO15504 / SPICE) covered in SENG 511 Software quality measurement standards (e.g., ISO 9126) explained here far@ucalgary.ca

25 ISO 9126: Parts ISO/IEC : Software engineering i -- Product quality -- Part 1: Quality model ISO/IEC TR :2003 Software engineering -- Product quality -- Part 2: External metrics ISO/IEC TR :2003 Software engineering -- Product quality -- Part 3: Internal metrics ISO/IEC TR :2004 Software engineering -- Product quality -- Part 4: Quality in use metrics far@ucalgary.ca

26 Definition Software Quality Characteristics ISO 9126: A set of attributes of a software product by which its quality is described and evaluated. A software quality characteristic may be refined into multiple levels of sub- characteristics. ti far@ucalgary.ca

27 ISO 9126: Quality Lifecycle Internal metrics do not rely on software execution ( static measures) External metrics are applied to running software Quality-in-use in use metrics are applied when the final product is used in real conditions Ideally, internal quality determines external quality and external quality determines quality in use. Example: Failure density Failure intensity Availability far@ucalgary.ca

28 ISO 9126: Part 1: Quality model ISO/IEC : 1: (Released in 2001) Information technology - Software quality characteristics & metrics - Part 1: Quality characteristics and subcharacteristics. This part provides the concepts introduced in the original standard is a recommended quality model which categorizes software quality in six characteristics, which are further subdivided into subcharacteristics. The subcharacteristics have been moved from the annex to become part of the standard. They have been reworded and several new ones added. There is also a definition of quality in use which defines the user s s view as a result of using the software. far@ucalgary.ca 28

29 ISO 9126: Part 2: External metrics ISO/IEC : 2: Information technology - Software quality characteristics & metrics - Part 2: External metrics. This part provides external metrics for measuring software quality characteristics. An external metric is a quantitative scale and measurement method, which can be used for measuring an attribute or characteristic of a software product, derived from the behaviour of the system of which it is a part. External metrics are applicable to an executable software product during testing or operating in later stage of development and after entering to operation process. far@ucalgary.ca 29

30 ISO 9126: Part 3: Internal metrics ISO/IEC : Information technology - Software quality characteristics & metrics - Part 3: Internal metrics. This part provides internal metrics for measuring software quality characteristics. An internal metric is a quantitative scale and measurement method, which can be used for measuring an attribute or characteristic of a software product, derived from the product itself, either direct or indirect. Internal metrics are applicable to a non executable software product during designing and coding in early stage of development process. far@ucalgary.ca 30

31 ISO 9126: Part 4: Quality-in- Use /1 Effectiveness The capability of the software product to enable users to achieve specified goals with accuracy and completeness in a specified context of use. Productivity The capability of the software product to enable users to expend appropriate amounts of resources in relation to the effectiveness achieved in a specified context of use. Safety The capability of the software product to achieve acceptable levels of risk of harm to people, business, software, property or the environment in a specified context of use. Satisfaction The capability of the software product to satisfy users in a specified context tof use. far@ucalgary.ca

32 Quality Model: ISO 9126 /1 Specifying i software quality for a product yet to be developed d is difficult for the customer and/or supplier. The customer needs to understand and communicate requirements for the product to be developed. The supplier needs to understand the requirement and to assess with confidence whether it is possible to provide the product with the right level of quality. ISO 9126 will serve to eliminate any misunderstanding between customer and supplier. ISO 9126 is the software product evaluation standard. This international standard defines six characteristics that describe, with minimal overlap, software quality. far@ucalgary.ca 32

33 Quality Model: ISO 9126 /2 1. Functionality is the set of attributes that bear on the existence of a set of functions and their specified properties. The functions are those that satisfy stated or implied needs. 2. Reliability is the set of attributes that bear on the capability of software to maintain its level of performance under stated conditions for a stated period of time. 3. Usability is the set of attributes that bear on the effort needed for use, and on the individual assessment of such use, by a stated or implied set of users. 4. Efficiency is the set of attributes that bear on the relationship between the level l of performance of the software and the amount of resources used, under stated conditions. 5. Maintainability is the set of attributes that bear on the effort needed to make specified modifications. 6. Portability is the set of attributes that bear on the ability of software to be transferred from one environment. far@ucalgary.ca 33

34 Quality Model: ISO 9126 /3 1 : n relation between Characteristics and Attributes (Sub- Characteristics) Characteristics Attributes Functionality Suitability Interoperability Accuracy Compliance Security Reliability Maturity Recoverability Fault tolerance Crash frequency Usability Understandability Learnability Operability Efficiency ce cy Time ebehaviour avou Resource behaviour Maintainability Analyzability Stability Changeability Testability Portability Adaptability Installability Conformance Replacability far@ucalgary.ca 34

35 ISO 9126 Views Custo omer Users s Supp plier Managers Developers far@ucalgary.ca

36 ISO 9126: Users View Users are mainly interested t din using the software, its performance and the effects of using the software. Users evaluate the software without knowing the internal aspects of the software, or how the software is developed. Users questions may include: Are the required functions available in the software? How reliable is the software? How efficient is the software? Is the software easy to use? How easy is it to transfer the software to another environment? How easy is to change this software? far@ucalgary.ca 36

37 ISO 9126: Developers View The process of development requires the user and the developer to use the same software quality characteristics, since they apply to requirement and acceptance. When developing off-the-shelf hlfsoftware, the implied lidneeds must be reflected in the quality requirement. Since developers are responsible for producing software which h will satisfy quality requirements they are interested in the intermediate product quality as well as in the final product quality. In order to evaluate the intermediate product quality at each phase of the development cycle, the developers have to use different metrics for the same characteristics because the same metrics are not applicable to all phases of the cycle. far@ucalgary.ca 37

38 ISO 9126: Managers View A manager is typically more interested t in the overall quality rather than in a specific quality characteristic, and for this reason will need to assign weights, reflecting business requirements, to the individual characteristics. The manager may also need to balance the quality improvement with management criteria i such as schedule delay or cost overrun, because he/she wishes to optimise quality within limited cost, human resources and time-frame. far@ucalgary.ca 38

39 ISO 9126: Summary The proposed quality model is very comprehensive, but it s not fully clear from the definitions how to precisely distinguish subcategories for many of the subcharacteristics the suggested measures are either trivial or hard to apply far@ucalgary.ca

40 ISO 9126 : Future A new series of standards is currently under development called Software Product Quality Requirements and Evaluation (SQuaRE - ISO 25000) This series of standards d will replace ISO 9126 and standards Note: the new standard will replace the word metric by measure far@ucalgary.ca

41 Software Quality: Measuring Customer Satisfaction

42 TQM: Customer Satisfaction Studies show that It is five times more costly to recruit a new customer than it is to keep an old customer and Dissatisfied customers tell 7 to 20 people about their experiences, while satisfied customers tell only 3 to 5 far@ucalgary.ca 42

43 TQM: Customer Satisfaction Total quality management (TQM) is aimed at long-term business success by linking gquality with customer satisfaction. With ever-increasing market competition, customer focus is the only way to retain the existing i customers and to expand market share. far@ucalgary.ca 43

44 Methods to Collect Data /1 How to collect data from customers? Three common methods to gather data: Personal face-to-face interviews Tl Telephone it interviewsi Mail questionnaires (self-administered) All require a kind of sampling to select a subset of the total customers. far@ucalgary.ca 44

45 Methods to Collect Data /2 Comparison of the data gathering techniques In Person Type of survey Interview Phone Mail Cost Sampling Response Rate Speed Reliability Observations Length of Interview Exhibits Validity = Disadvantage -- = Worst + = Advantage ++ = Best +- = Could be either an Advantage or Disadvantage Figure from Kan s Book far@ucalgary.ca 45

46 Sampling Methods For any survey, the sampling design is of utmost importance in obtaining unbiased, representative data. 4 basic types of probability sampling: 1. Simple random sampling 2. Systematic sampling 3. Stratified sampling 4. Cluster sampling far@ucalgary.ca 46

47 1. Simple Random Sampling Asampleofsizen size n is drawn from a population in such a way that every possible sample of size n has the same chance of being selected. To take a simple random sample, each individual in the population must be listed once and only once. A mechanical procedure (i.e., using a random number table, or random number generating program) is used to draw the sample. To avoid repeated drawing of the same individual, the sampling is done without replacement. On each successive draw the probability of an individual being selected increases slightly because there are fewer and fewer individuals left unselected from the population. If, on any given draw, the probabilities are equal of all remaining items being selected, then we have a simple random sample. far@ucalgary.ca 47

48 2. Systematic Sampling In systematic ti sampling instead of using a table of random numbers, one simply goes down a list taking every kth individual, starting with a randomly selected case among the first k individuals. k is the ratio between the size of the population and the size of fthe sample to be drawn. (1/k : sampling fraction) Example: draw a sample of 500 customers from a population of 20, k = 20000/500 = 40 Starting with a random number between 1 and 40 (say, 18), then we would draw every fortieth th on the list (58, 98, 138, ). far@ucalgary.ca 48

49 3. Stratified Sampling In a stratified sample items are classified into nonoverlapping groups, called strata, and then simple random samples are selected from each stratum. The strata are usually formed dbased on important variables pertaining to the parameter of interest. Example: customers with high-speed internet systems may have a set of satisfaction criteria for software products that is different from those who have low-speed systems. A stratified sample should include customer type as one of the stratification variables. Stratified samples can be designed to yield greater accuracy for the same cost, or for the same accuracy with less cost. far@ucalgary.ca 49

50 4. Cluster Sampling Sometimes it is much more cost effective to divide the population into a large number of groups, called clusters, and to sample among the clusters. A cluster sample is a simple random sample in which heach sampling unit is a cluster of elements. Usually geographical units such as cities, districts, schools, or work plants are used as units for cluster sampling. Example: if a company has many branch offices throughout the world and an in-depth face-to-face interview with a sample of its customers is desired, then a cluster sample using branch offices as clusters (of customers) may be the best sampling approach. far@ucalgary.ca 50

51 Sample Size /1 How large a sample is sufficient? i The answer depends on the confidence level we want and dthe margin of error we can tolerate. t How to bind sample size with customer satisfaction? Smaller error margin Larger sample size Higher level of confidence far@ucalgary.ca 51

52 Sample Size /2 Sample size (n) forsimple random sampling: 2 NZ p 1 p n 2 2 NB Z p 1 p N : population size Z : Z statistic from normal distribution: for 80% confidence level, Z = 1.28 for 85% confidence level, Z = 1.45 for 90% confidence level, Z = 1.65 for 95% confidence level, Z = 1.96 p : estimated satisfaction level B : margin of error far@ucalgary.ca 52

53 Sample Size /3 Sample sizes for 10, customers for 80% - 95% levels of confidence (Z) and 5% and 3% margin of error (B) Larger sample size Expected Customer 80% confidence 85% confidence 90% confidence 95% confidence Satisfaction (p) ±5% ±3% ±5% ±3% ±5% ±3% ±5% ±3% 80% % % % Smaller Sample size far@ucalgary.ca 53

54 Example Suppose that you are going to design a questionnaire to be sent to customers who bought your software product. The population size is 10,000 customers and margins of error is supposed to be 5%. a) Calculate the sample size for the expected customer satisfaction of 90% and confidence factor of 80%. N = 10,000 Z = 1.28 for 80% confidence level p = B = NZ p 1 p 10, n NB Z p 1 p 10, n 59 far@ucalgary.ca 54

55 Example (cont d) b) Suppose that t you change the sample size to n=136 and the other parameters except the confidence level remain unchanged. Calculate the new confidence level. N = 10, p = n = 136 B = 0.05 Z =? 2 NZ 2 p 1 p 10, Z NB Z p 1 p 10, Z Z Z Confidence level is 95% for this value ofz far@ucalgary.ca 55

56 Question: How Good is Good Enough? How much customer satisfaction is good enough? Should my company invest $2,000,000 to improve satisfaction from 85% to 90%? Given that my company s customer satisfaction is at 95%, should I invest another million dollars to improve it further or should I do this later? far@ucalgary.ca 56

57 Babich s Study /1 Babich (1992) Model based on a simplified model of customer satisfaction and market share that contains only three companies: A, B, and C. When customers are dissatisfied with company A, they choose company B or C, etc. A t A t A t At 1 At 1 x Bty Ctz G At Ct At Bt At Bt Ct B t B t B t Bt 1 Bt 1 y Ax t Ctz G Bt Ct At Bt At Bt Ct C t C t C t Ct 1 Ct 1 z Ax t Bty G Bt Ct At Ct At Bt Ct far@ucalgary.ca 57

58 Babich s Study /2 A : number of fa customers B : number of B customers C : number of C customers G : number of new customers to market x : dissatisfaction level with A products y : dissatisfaction level with B products z : dissatisfaction level with C products t : time far@ucalgary.ca 58

59 Babich s Study /3 Market shares of fthe three companies assuming satisfaction levels of 95%, 91%, and 90% for A, B, and C, respectively, over a number of time periods and equal initial market share. After 12 time periods the 95% satisfaction product (company A) would basically own the market. Figure from Kan s Book far@ucalgary.ca 59

60 Babich s Study /4 If the satisfaction i levels l of companies B and C were 98% and 99%, respectively, and company A s satisfaction level remained at 95%, company A s product would have less than 10% market share in 24 time periods. Example from Kan s Book far@ucalgary.ca 60

61 Answer: How Good is Good Enough? From Bbih Babich ssimple i model and examples the answer is: You have to be better than your competitors! Therefore, it is important to measure not only one s customer satisfaction level, but also the competitors satisfaction level. l Indeed, many companies have been doing exactly that. t far@ucalgary.ca 61

62 Software Quality Management

63 Demming s 14 Quality Points 1. Create constancy of purpose for improvement of product or service. 2. Adopt the new philosophy. 3. Cease dependence on inspection to achieve quality. 4. End the practice of awarding business on the basis W E Deming of price tag alone. Instead, minimize total cost by working with a single supplier. 5. Improve constantly tl and forever every process for planning, production and service. 6. Institute training on the job. 7. Adopt and institute leadership. W. E. Deming ( ) The Father of Modern Quality far@ucalgary.ca 63

64 Demming s 14 Quality Points 8. Drive out fear. 9. Break down barriers between staff areas. 10. Eliminate slogans, exhortations, and targets for the work force. 11. Eliminate numerical quotas for the work force and numerical goals for management. 12. Remove barriers that rob people of pride of workmanship. Eliminate the annual rating or merit system. 13. Institute a vigorous program of education and selfimprovement for everyone. 14. Put everybody in the company to work to accomplish the transformation. W. E. Deming ( ) The Father of Modern Quality far@ucalgary.ca 64

65 TQM: Total Quality Management TQM is a style of management aiming at achieving long-term term success by linking quality with customer satisfaction Other variations: Other variations: Studying customer needs gather Total Quality Control (HP) Market Driven Quality (IBM) Experience Factory (Basili) Studying customer needs, gather customer requirements, measuring customer satisfaction Ahi Achieve business and product development process improvement Creation of company-wide quality culture Figure from Kan s Book far@ucalgary.ca 65

66 Practice of TQM: Essentials Focus on quality Cooperate with customers Continuously improve development process Encourage constructive critics and empower employees Use the problem solving/problem prevention cycle Use measurements to back decisions 66

67 References Standards: [IEEE730] IEEE : IEEE Standard for Software Quality Assurance Plans, Institute of Electrical and Electronics Engineers, 29-Sep [ISO8402] ISO : Quality management and quality assurance Vocabulary, [ISO9126] ISO/IEC : Software engineering -- Product quality -- Parts 1-4, ed. 1, Current stage JTC 1/SC 7, Software Quality: [BBL76] Boehm, Barry W., Brown, J. R, and Lipow, M.: Quantitative evaluation of software quality, International Conference on Software Engineering, Proceedings of the 2nd international conference on Software engineering, [BBK+78] Boehm, B. W., Brown, J. R., Kaspar, H.,Lipow, M., McLeod, G., and Merritt, M., Characteristics of Software Quality, North Holland, [Kan02] Stephen H. Kan: Metrics and Models in Software Quality Engineering, 2 nd Edition, Addison-Wesley, [MRW77] McCall, J. A., Richards, P. K., and Walters, G. F., "Factors in Software Quality", Nat'l Tech. Information Service, no. Vol. 1, 2 and 3, Stephen H. Kan, Metrics and Models in Software Quality Engineering, 2 nd Edition, Addison-Wesley, P. Babich, Customer Satisfaction: How Good Is Good Enough? Quality Progress, December 1992, pp far@ucalgary.ca