Software Quality Metrics. Analyzing & Measuring Customer Satisfaction (Chapter 14)

Transcription

1 Software Quality Metrics Analyzing & Measuring Customer Satisfaction (Chapter 14) By Zareen Abbas Reg# 169/MSSE/F07 Usman Thakur Reg# 181/MSSE/F07 1

2 Overview-Quality Product quality and customer satisfaction together form the total meaning of quality. Customer satisfaction is the ultimate validation of quality. Quality is combination of two things: 1. Product Quality and 2. Customer Satisfaction Small q and Big Q 2

3 Overview-Quality 3

4 Overview-Quality Customer buy not product but assurance Producer sell their assurance not product TQM-Total Quality Management sole focus is linking quality with customer satisfaction 4

5 What is Customer Segment Every business must decide "what market it is in" - what products and services it offers, to whom they will be offered, in which geographic area and which firms are its competitors Once these strategic decisions are made, the business can refine "to whom it offers" its products and services into a set of customer or market segments 5

6 Process Model Approach 6

7 Why Customer Satisfaction 7

8 Why Customer Satisfaction Many studies suggested that enhancing customer satisfaction is the bottom line of business success: Ever-increasing market competition Only way to retain the Customers To expand market share To gain more profit To enhance/ improve product satisfaction level eg. 90% to 95% Studies show that it is five times more costly to recruit a new customer than it is to keep an old customer: Why is it costly? 8

9 Why Customer Satisfaction It is fact that dissatisfied customers tell 7 to 20 people about their experiences While satisfied customers tell Only 3 to 5 people 9

10 10

11 ISO Quality Policy and Customer Focus 11

12 tings\administrator\desktop\prest Customer satisfaction\customer Metrics 411 Customer Satisfaction gooood.mht 12

13 Customer Satisfaction is the crucial requirement for a good reputation. Those who have it, will find all doors open But those who have lost it, frequently perish or have to struggle hard to have their names associated with that magic word Quality 13

14 Customer Satisfaction Total customer satisfaction is the primary quality issue. Customers are the only people who can determine total customer satisfaction. To achieve total customer satisfaction, the organization must know the customer, itself, its product, and its competition. 14

15 15

16 Metrics for Customer Satisfaction Surveys 16

17 Metrics for Customer Satisfaction Surveys TQM, more and more companies are conducting surveys to measure their customers' satisfaction. We will discuss customer satisfaction surveys and the analysis of survey data. As an example, we describe an analysis of the relationship between overall customer satisfaction and satisfaction with specific attributes for a software product. 17

18 Why to Surveys for Customer Satisfaction There are various ways to obtain customer feedback with regard to their satisfaction levels with the products and the company. Like, telephone follow-up regarding a customer's satisfaction at a regular time after the purchase is a frequent practice by many companies. Other sources include customer complaint data, direct customer visits, customer advisory councils, user conferences, and the like. To obtain representative and comprehensive data, representative of the entire customer base. 18

19 Methods of Survey Data Collection There are three common methods to gather survey data: 1. Face-to-face interviews (Personal) 2. Telephone interviews 3. Mailed questionnaires The personal interview method requires the interviewer to ask questions based on a prestructured questionnaire and to record the answers. The primary advantage of this method is the high degree of validity of the data. 19

20 Advantages and Disadvantages of Three Survey Methods 20

21 Sampling Methods When the customer base is large, it is too costly to survey all customers. Estimating the satisfaction level of the entire customer population through a representative sample is more efficient. To obtain representative samples, scientific probability sampling methods must be used. There are basic types of probability sampling: Simple random sampling, Systematic sampling, Cluster (group) sampling. 21

22 How large a sample is sufficient? The answer to this question depends on the confidence level we want and the margin of error we can tolerate. The higher the level of confidence we want from the sample estimate, and the smaller the error margin, the larger the sample we need, and vice versa. For each probability sampling method, specific formulas are available for calculating sample size, some of which (such as that for cluster sampling) are quite complicated. The following formula is for the sample size required to estimate a population proportion (e.g., percent satisfied) based on simple random sampling: 22

23 Formula for sampling 23

24 Formula for sampling N: Population = 10,000 Z = confidence level 90 % = 1.65 p = satisfaction level = 80 % B: margin of error = 5 % n = 30,000*(1.65) 2 x 80(1-80). 30,000*5 2 x [ x 80(1-80)] n =??? 24

25 Sample Size (for 10,000 customers) in Relation to Confidence Level and Error Margin 25

26 Sample Size (for 10,000 customers) in Relation to Confidence Level and Error Margin 26

27 Sample Size (for 10,000 customers) in Relation to Confidence Level and Error Margin 27

28 Sample Size (for 10,000 customers) in Relation to Confidence Level and Error Margin 28

29 Sample Size (for 10,000 customers) in Relation to Confidence Level and Error Margin 29

30 Analyzing Satisfaction Data 30

31 Analyzing Satisfaction Data Customer satisfaction is often measured by customer survey data via the five-point scale: Very satisfied Satisfied Neutral Dissatisfied Very dissatisfied. 31

32 Analyzing Satisfaction Data Satisfaction with the overall quality of the product and its specific dimensions is usually obtained through various methods of customer surveys. For example, the specific parameters of customer satisfaction by IBM include CUPRIMDA: Capability Functionality Usability Performance Reliability Installability Maintainability Documentation/information Availability HP include FURPS: functionality, usability, reliability, performance, and service 32

33 Analyzing Satisfaction Data The data are usually summarized in terms of percent satisfied. In presentation, run charts or bar charts to show the trend of percent satisfied are often used. We recommend that confidence intervals be formed for the data points so that the margins of error of the sample estimates can be observed immediately 33

34 Quarterly Trend of Percent Satisfied with a Hypothetical Product Traditionally, the 95% confidence level is used for forming confidence intervals and the 5% probability (p value) is used for significance testing. 34

35 Quarterly Trend of Percent Satisfied with a Hypothetical Product Traditionally, the 95% confidence level is used for forming confidence intervals and the 5% probability (p value) is used for significance testing. 35

36 Quarterly Trend of Percent Satisfied with a Hypothetical Product Traditionally, the 95% confidence level is used for forming confidence intervals and the 5% probability (p value) is used for significance testing. 36

37 Approaches for Analyzing Satisfied Non-satisfied Usually the 1 st metric, percent satisfaction, is used. In practices that focus on reducing the percentage of nonsatisfaction, much like reducing product defects, non-satisfied metric is used. 37

38 Non-satisfied metrics Although percent satisfied is perhaps the most used metric, some companies, such as IBM, choose to monitor the inverse, the percent non-satisfied. Non-satisfied includes the neutral dissatisfied, and very dissatisfied in the five-point scale. The rationale to use percent non-satisfied is to focus on areas that need improvement. This is especially the case when the value of percent satisfied is quite high. Figure 12.3 in Chapter 12 shows an example of IBM Rochester's percent non-satisfied 38

39 In addition to forming percentages for various satisfaction or dissatisfaction categories, the weighted index approach can be used. For instance, some companies use the Net Satisfaction Index (NSI) to facilitate comparisons across product. The NSI has the following weighting factors: Completely satisfied = 100% Satisfied = 75% Neutral = 50% Dissatisfied = 25% Completely dissatisfied = 0% 39

40 Specific Attributes and Overall Satisfaction Co-relationship between Attributes and overall satisfaction 40

41 Specific Attributes and Overall Satisfaction The major advantage of monitoring customer satisfaction with specific attributes of the software, in addition to overall satisfaction, is that such data provide specific information for improvement. The profile of customer satisfaction with those attributes (e.g., CUPRIMDA) indicates the areas of strength and weakness of the software product. One easy mistake in customer satisfaction analysis, however, is to equate the areas of weakness with the priority of improvement, and to increase investment to improve those areas. Example Documentation and Reliability 41

42 On the other hand, customers may not like the product's documentation. To answer the question on priority of improvement, therefore, the subject must be looked at in the broader context of overall customer satisfaction with the product. Specifically, the correlations of the satisfaction levels of specific attributes with overall satisfaction need to be examined. After all, it is the overall satisfaction level that the software developer aims to maximize 42

43 Variables Dependent variable : Overall customer satisfaction Independent variables: Attribute (Reliability) 43

44 How to see relationship of attributes with overall satisfaction Two statistical approaches Least-squares multiple regression (Ordinal) Logistic regression 44

45 Logistic Regression In statistics, logistic regression (sometimes called the logistic model) is used for prediction of the probability of occurrence of an event by fitting data to a logistic curve. It makes use of several predictor variables that may be either numerical or categorical. For example, the probability that a person has a heart attack within a specified time period might be predicted from knowledge of the person's age, sex and body mass index. Logistic regression is used extensively in the medical and social sciences as well as marketing applications such as prediction of a customer's propensity to purchase a product or cease a subscription. 45

46 Logistic Regression For the logistic regression approach, we classified the five-point scale into a dichotomous variable (categorizes data into two groups) 1. Very Satisfied 2. Satisfied 1,2,3 = 0 3. Neutral vs 4. Dissatisfied 4,5 = 1 5. Very dissatisfied Very satisfied and satisfied (4 and 5) versus non-satisfied (1, 2, and 3). Categories 4 and 5 were recoded as 1 and categories 1, 2, and 3 were recoded as 0. The dependent variable, therefore, is the odds (probability) ratio of satisfied and very satisfied versus non-satisfied. DV: A variable that categorizes data into two groups such as bankrupt versus solvent or energy company versus not energy company. Also called Dummy Variable 46

47 Logistic Regression The odds ratio is a measurement of association that has been widely used for categorical data analysis. 47

48 Ordinary regression Both approaches overall customer satisfaction is the dependent variable, and satisfaction levels with UPRIMD-A are the independent variables. The purpose is to assess the correlations between each specific attribute and overall satisfaction simultaneously. For the ordinary regression approach, we use the original fivepoint scale. The scale is an ordinal variable. Sensitivity research in the literature, however, indicates that if the sample size is large (such as in our case), violation of the interval scale and the assumption of Gaussian distribution results in very small bias. In other words, the use of ordinary regression is quite robust for the ordinal scale with large samples. 48

49 Analysis: Ordinary regression 49

50 Analysis: Ordinary regression 50

51 Analysis: Ordinary regression 51

52 Results of Ordinal Regression Analysis 52

53 Usability with reliability, installability, and documentation, and performance with availability are the more notable ones. In relation to overall satisfaction, reliability, usability, and installability have the highest correlations. Reliability being the most significant variable implies that although customers are quite satisfied with the software's reliability (93.8%), reliability is still the most determining factor for achieving overall customer satisfaction. 53

54 Results of the Logistic Regression model The most striking observation is that the significance of availability in affecting customer satisfaction is in contrast to findings from the ordinary regression analysis. Now availability ranks third, after reliability and usability, in affecting overall satisfaction. The difference observed from the two models lies in the difference in the scaling of the dependent and independent variables in the two approaches. Combining the two findings, we interpret the data as follows: 54

55 Results of Logistic Regression Analysis 55

56 Combining the two findings, we interpret the data as follows: Availability is not very important in influencing the average shift in overall customer satisfaction from one level to the next (from dissatisfied to neutral, from neural to satisfied, etc.). However, availability is very important in affecting whether customers are satisfied versus nonsatisfied. Therefore, availability is a sensitive factor in customer satisfaction and should be improved despite its level of high satisfaction. 56

57 2 nd Part 57

58 Conditional Probabilities Dependent variable of the logistic regression model (satisfied versus non-satisfied) is more appropriate for our purpose, we use the results of the logistic model for the rest of our example 58

59 Conditional Probabilities 59

60 Logistic Regression coefficients versus satisfaction Levels 60

61 Satisfaction with Company Technical solutions: quality/reliability, availability, ease of use, pricing, installation, new technology Support and service: flexible, accessible, product knowledge Marketing: solution, central point of contact, information Administration: purchasing procedure, billing procedure, warranty expiration notification Delivery: on time, accurate, post delivery process Company image: technology leader, financial stability, executives image. 61

62 Satisfaction with Company It is apparent that customer satisfaction at both the company level and the product level needs to be analyzed and managed. Knowledge about the former enables a company to take a comprehensive approach to total quality management; knowledge about the latter provides specific clues for product improvements. Yet another type of analysis centers on why customers choose a company's products over other companies', and vice versa. This kind of analysis requires information that is not available from regular customer satisfaction surveys, be they product level or company level. It requires data about customers' decision making for purchases and requires responses from those who are not the company's current customers as well as those who are. This type of analysis, albeit difficult to conduct, is worthwhile because it deals directly with the issue of gaining new customers to expand the customer base. 62

63 How good is good enough Should be 100% for long term. Should consider competitors. Even if there are no competitors unsatisfied customers would look for different alternatives. That becomes a trigger for emerging competitions. So total customer satisfaction should be the focus. Should my company invest $2,000,000 to improve satisfaction from 85% to 90%? Given that my company's customer satisfaction is at 95%, should I invest another million dollars to improve it or should I do this later? 63

64 Satisfaction level vs Market share Algorithm for market share. 64

65 Example Comparison: 65

66 the answer to the "how good is good enough" is obvious: You have to be better than your competitors. Therefore, it is important to measure not only one's customer satisfaction level, but also the satisfaction level of one's competitors. Indeed, many companies have been doing exactly that. Do Benchmarking 66

67 Recommendations! Total quality management (TQM) literature recommends three key elements of a good customer satisfaction management program: a post purchase call back program, a complaint management process. And a customer satisfaction survey program. 67

68 For Small Organizations For small organizations that don't have a customer focus program in place, It is recommended to integrate one or more of these practices into their development process. Good requirements. Design FVT (Functional Verification Test) Customer like Testing Internal Beta Customer Beta Validation and feedback for further improvement Special Inbound customer testing. 68

69 For Small Organizations Requirements: Good requirements-gathering and analysis techniques always involve the users closely. For long-term product strategy, customer advisory councils usually provide good directions For Extreme Programming (XP), customer involvement is regarded as so important that an onsite customer is part of the development team and is responsible for domain expertise and acceptance testing. For object-oriented processes in general, the role of use cases in requirements analysis is another illustration of the importance of user involvement. Design: The prototyping approach is a good way to get customers' feedback and to ensure that the product satisfies requirements. In recent years, some companies use the term user-centered design (UCD) to signify the heavy user involvement in their design process. 69

70 For Small Organizations Functional verification test (FVT): Customer testing can start early, when FVT is near completion. This early customer testing is not a beta program, and is called a hackers' invitational or customer invitational program. A selected small number of customer experts who know the product well are invited to the vendor's development laboratory to test the new function and to attempt to break the system. Customer like testing: Unit tests and functional verification tests are normally for functional defect removal. For product-level testing or system testing, which are normally conducted by an independent test team, many software organizations adopt a customer like testing strategy. The testing environments and the test scenarios simulate customer operations, and therefore offer the chance to find latent defects that are likely to be encountered by customers. 70

71 For Small Organizations Internal Beta, or eating one's own cooking: When product development is about complete, and independent testing about to start, the development organization can put the product into production mode running its own business operations. This strategy puts the development team into the customer's shoes and can really motivates the team's quality focus. Customer beta: This is a broad-based customer validation program. It usually starts when final testing is about done and before the product is released to the market. The product has to be at a fairly good quality level at this stage. The main purpose is to get customer validation and feedback for further improvement. Customer beta programs should not be used as an extended development phase as a major activity for defect removals. Special inbound customer testing: Unlike the broad-based beta programs, this strategy is to invite a small set of one's core customers to the development laboratory to test applications before releasing the product to the market. This approach focuses on depth, and is relevant for a small set of customers who depend on the software for their mission-critical business applications 71

72 Conclusion Finally, we emphasize that measuring and analyzing customer satisfaction is but one element of customer satisfaction management. A good customer satisfaction management process must form a closed loop of measurement, analysis, and actions. While it is not the intent of this chapter to cover the customer satisfaction management process, we recommend that such a process cover at least the following elements: Measure and monitor the overall customer satisfaction over time, one's own as well as key competitors'. Perform analyses on specific satisfaction dimensions, quality attributes of the products and their strengths, weaknesses, prioritization, and other relevant issues. Perform root cause analysis to identify inhibitors for each dimension and attribute. Set satisfaction targets (overall and specific) by taking competitors' satisfaction levels into consideration. Formulate and implement action plans based on the above. 72

73 Extra 73

74 Outcomes of measuring customer satisfaction Measuring customer satisfaction provides you with valuable information from the one group most critical to your business success - your customers. If customer satisfaction surveys are conducted accurately, you should be able to discover the following key insights: 1. Specific areas where customer service improvements are necessary 2. Expectations of your customers regarding customer service 3. Specific reasons for customer dissatisfaction 4. How to improve retention of existing customers 5. How to make customers feel valued 6. Missed opportunities to demonstrate your capacity to solve problems 74

75 Why Include Customers of Competitors The rationale for including non-customer (customers of competitors) benchmarking is that without the noncustomer data customer satisfaction levels are arbitrary. Both sets of data allow an organization to exploit its strengths, and put initiatives in place to narrow and eliminate any gaps between expectations and performance. In fact, the best performing organizations benchmark themselves against: their best competitor the industry average a world class supplier in a similar industry 75

76 How Exit Interviews Can Translate into Huge Profit Increases By measuring only customer satisfaction levels, organizations miss former customers who have left... because they no longer had the need for, or were unhappy with the products or services being offered. Measuring customer retention, on the other hand, relates directly to the bottom line. Long term customers spend more, refer new clients and are less costly to do business with. Ironically, past customers present every company with an opportunity. They can tell the organization exactly what parts of the business to fix in order to reduce the number of customers at risk. This improves customer retention and, subsequently, profitability. An average organization loses about 15% of its customers every year. But if this can be reduced to 10%, bottom line profits improve 35% to 85%. Finding out why customers leave can often be difficult since the majority of unhappy customers don't complain, they simply quit. Exit interviews solve this problem. 76

77 Achieving Continuous Quality Improvement Continuous quality improvement begins by identifying customer expectations for all key "moments of truth" - the critical interactions customers have with the organization. The best way to understand customer expectations is to listen to customers using qualitative research techniques. This usually requires skillful probing by someone practiced in customer satisfaction measurement. After identifying expectations, customer satisfaction can readily be measured, how? However, this requires the customer to answer specific questions about how he or she feels about the company's performance. This is why it is so important to capture their interest and build the credibility needed to gain their cooperation. The task is made considerably easier by speaking the customer's language and presenting only issues that are truly significant. 77

78 Customer Focus 78

79 A Predictor of Financial Performance Measured the right way, customer satisfaction is a leading indictor of a company s financial performance. Managed scientifically using a proven, standard metric, customer satisfaction drives loyalty, profitability and growth. The American Customer Satisfaction Index (ASCI) is one of the best-in-breed solutions for customer satisfaction measurement that is tied directly to financial performance. 79

80 Identify customer needs and expectations The identification of customer needs and expectations requires systematic, thorough, and continuous communication. The most important aspect of this process is to listen to the customer. Once customer needs are identified, these needs must be monitored continuously to ensure that the product and/or service still satisfies them. 80

81 Develop customer relationships Customer relationships are the core of customerdriven project management. This relationship ensures continuous customer satisfaction. Customer relationships require communication, support, and responsiveness. Trust is absolutely essential for any relationship and is necessary to ensure full participation of all members on the customer-driven team. 81

82 Thank you Q & A 82