Measure Phase Measurement System Analysis

Transcription

1 Measure Phase Measurement System Analysis

2 Measurement System Analysis Welcome to Measure Process Discovery Six Sigma Statistics Measurement System Analysis Basics of MSA Variables MSA Attribute MSA Process Capability Wrap Up & Action Items 2

3 Introduction to MSA We have learned the heart and soul of Six Sigma is data. How do you know the data you are using is accurate and precise? How do know if a measurement is a repeatable and reproducible? How good are these? Measurement System Analysis or MSA 3

4 Measurement System Analysis MSA is a mathematical procedure to quantify variation introduced to a process or product by the act of measuring. Item to be Measured Operator Reference Measurement Process Equipment Measurement Environment Procedure The item to be measured can be a physical part, document or a scenario for customer service. Operator can refer to a person or can be different instruments measuring the same products. Reference is a standard that is used to calibrate the equipment. Procedure is the method used to perform the test. Equipment is the device used to measure the product. Environment is the surroundings where the measures are performed. 4

5 Measurement Purpose In order to be worth collecting measurements must provide value - that is, they must provide us with information and, ultimately, knowledge. The question What do I need to know? must be answered before we begin to consider issues of measurements, metrics, statistics or data collection systems. Too often organizations build complex data collection and information management systems without truly understanding how the data collected and metrics calculated actually benefit the organization. 5

6 Purpose The purpose of MSA is to assess any error due to inaccuracy of our measurement systems. The error can be partitioned into specific sources: Precision Repeatability - within an operator or piece of equipment Reproducibility - operator to operator or attribute gage to attribute gage Accuracy Stability - accuracy over time Linearity- accuracy throughout the measurement range Resolution how detailed is the information Bias Off-set from true value Constant Bias Variable Bias typically seen with electronic equipment, amount of Bias changes with setting levels 6

7 Accuracy and Precision Accurate but not precise - On average these shots are in the center of the target but there is a lot of variability Precise but not accurate - The average is not on the center but the variability is small 7

8 MSA Uses MSA can be used to ~ Compare internal inspection standards with the standards of your customer. Highlight areas where calibration training is required. Provide a method to evaluate inspector training effectiveness as well as serve as an excellent training tool. Provide a great way to: Compare existing measurement equipment. Qualify new inspection equipment. 8

9 Why MSA? Measurement System Analysis is important to: Study the % of variation in our process caused by our measurement system. Compare measurements between operators. Compare measurements between two (or more) measurement devices. Provide criteria to accept new measurement systems (consider new equipment). Evaluate a suspect gage. Evaluate a gage before and after repair. Determine true process variation. Evaluate effectiveness of training program. 9

10 Appropriate Measures Appropriate Measures are: Sufficient available to be measured regularly Relevant help to understand/isolate the problems Representative - of the process across shifts and people Contextual collected with other relevant information that might explain process variability. Wadda ya wanna measure!?! 10

11 Poor Measures Poor Measures can result from: Poor or non-existent operational definitions Difficult measures Poor sampling Lack of understanding of the definitions Inaccurate, insufficient or non-calibrated measurement devices Measurement Error compromises decisions affecting: Customers Producers Suppliers 11

12 Examples of What to Measure Examples of what and when to measure: Primary and secondary metrics Decision points in Process Maps Any and all gauges, measurement devices, instruments, etc X s in the process Prior to Hypothesis Testing Prior to modeling Prior to planning designed experiments Before and after process changes To qualify operators MSA is a Show Stopper!!! 12

13 Components of Variation Whenever you measure anything the variation you observe can be segmented into the following components Observed Variation Unit-to-unit (true) Variation Measurement System Error Precision Accuracy Repeatability Reproducibility Stability Bias Linearity All measurement systems have error. If you do not know how much of the variation you observe is contributed by measurement system error you cannot make confident decisions. If you were one speeding ticket away from losing your license how fast would you be willing to drive on your local freeway? 13

14 Precision A precise metric is one that returns the same value of a given attribute every time an estimate is made. Precise data are independent of who measures them or when the measurement is made. Precision can be partitioned into two components: Repeatability Reproducibility Repeatability and Reproducibility = Gage R+R 14

15 Repeatability Repeatability is the variation in measurements obtained with one measurement instrument used several times by one appraiser while measuring the identical characteristic on the same part. Y For example: Repeatability Manufacturing: One person measures the purity of multiple samples of the same vial and gets different purity measures. Transactional: One person evaluates a contract multiple times (over a period of time) and makes different determinations of errors. 15

16 Reproducibility Reproducibility is the variation in the average of the measurements made by different appraisers using the same measuring instrument when measuring the identical characteristic on the same part. Reproducibility Y Operator A Operator B For example: Manufacturing: Different people perform purity test on samples from the same vial and get different results. Transactional: Different people evaluate the same contract and make different determinations. 16

17 Time Estimate Exercise Exercise objective: Demonstrate how well you can estimate a 10 second time interval. 1. Pair up with an associate. 2. One person will say start and stop to indicate how long they think the 10 seconds last. Do this six times. 3. The other person will have a watch with a second hand to actually measure the duration of the estimate. Record the value where your partner cannot see it. 4. Switch tasks with partner and do it six times also. 5. Record all estimates. What do you notice? 17

18 Accuracy An accurate measurement is the difference between the observed average of the measurement and a reference value. When a metric or measurement system consistently over or under estimates the value of an attribute it is said to be inaccurate Accuracy can be assessed in several ways: Measurement of a known standard Comparison with another known measurement method Prediction of a theoretical value What happens if we do not have standards, comparisons or theories? True Average Warning, on a cross country trip do not assume your gasoline gage is gospel. Accuracy Measurement 18

19 Accuracy Against a Known Standard In transactional processes the measurement system can consist of a database query. For example, you may be interested in measuring product returns where you will want to analyze the details of the returns over some time period. The query will provide you all the transaction details. However, before you invest a lot of time analyzing the data you must ensure the data has integrity. The analysis should include a comparison with known reference points. For the example of product returns the transaction details should add up to the same number that appears on financial reports, such as the income statement. 19

20 Accuracy versus Precision ACCURATE PRECISE BOTH + = Accuracy relates to how close the average of the shots are to the Master or bull's-eye. NEITHER Precision relates to the spread of the shots or Variance. 20

21 Bias Bias is defined as the deviation of the measured value from the actual value. Calibration procedures can minimize and control bias within acceptable limits. Ideally Bias can never be eliminated due to material wear and tear! Bias Bias 21

22 Stability Stability of a gage is defined as error (measured in terms of Standard Deviation) as a function of time. Environmental conditions such as cleanliness, noise, vibration, lighting, chemical, wear and tear or other factors usually influence gage instability. Ideally gages can be maintained to give a high degree of Stability but the issue can never be eliminated unlike Reproducibility. Gage Stability studies should be the first exercise after calibration procedures. Control Charts are commonly used to track the Stability of a measurement system over time. Drift Stability is Bias characterized as a function of time! 22

23 B i a s (y) Linearity Linearity is defined as the difference in Bias values throughout the measurement range in which the gauge is intended to be used. This tells you how accurate your measurements are through the expected range of the measurements. It answers the question "Does my gage have the same accuracy for all sizes of objects being measured?" Linearity = Slope * Process Variation % Linearity = Slope * Low * Nominal * * High Reference Value (x) y = a + b.x y: Bias, x: Ref. Value a: Slope, b: Intercept 23

24 Types of MSA s MSA s fall into two categories: Attribute Pass/Fail Go/No Go Document preparation Surface imperfections Customer Service response Variable Continuous scale Discrete scale Critical dimensions Pull strength Warp Transactional projects typically have Attribute based measurement systems. Manufacturing projects generally use Variable studies more often but do use Attribute studies to a lesser degree. 24

25 Variable MSA s SigmaXL calculates a column of variance components (VarComp) that are used to calculate % Gage R&R using the ANOVA Method. Measured Value True Value Estimates for a Gage R&R study are obtained by calculating the variance components for each term and for error. Repeatability, Operator and Operator*Part components are summed to obtain a total Variability due to the measuring system. We use variance components to assess the Variation contributed by each source of measurement error relative to the total Variation. 25

26 MSA Study Open the worksheet Gage AIAG2 - SigmaXL Format 26

27 27 Analyze Gage R&R Results

28 MSA Cheat Sheet Contribution of Variation to the total Variation of the study. % Contribution, based on variance components, is calculated by dividing each value in VarComp by the Total Variation then multiplying the result by 100. Use % Study Var when you are interested in comparing the measurement system Variation to the total Variation. % Study Var is calculated by dividing each value in Study Var by Total Variation and Multiplying by 100. Study Var is calculated as 5.15 times the Standard Deviation for each source. (5.15 is used because when data are normally distributed, 99% of the data fall within 5.15 Standard Deviations.) 28

29 MSA Cheat Sheet SigmaXL Report ~ When the process tolerance is entered in the system, SigmaXL calculates % Tolerance which compares measurements system Variation to customer specification. This allows us to determine the proportion of the process tolerance that is used by the Variation in the measurement system. Distinct Categories (Rounded Down ) 29

30 Number of Distinct Categories The number of distinct categories tells you how many separate groups of parts the system is able to distinguish. 1 Data Category Unacceptable for estimating process parameters and indices Only indicates whether the process is producing conforming or nonconforming parts 2-4 Categories Generally unacceptable for estimating process parameters and indices Only provides coarse estimates Recommended 5 or more Categories 30

31 Standards for Gage Acceptance % Tolerance or % Study Variance % Contribution System is 10% or less 1% or less Ideal 10% - 20% 1% - 4% Acceptable 20% - 30% 5% - 9% Marginal 30% or greater 10% or greater Poor 31

32 Creating a Components of Variation Chart 32

33 SigmaXL Graphic Output Cheat Sheet Components of Variation The SigmaXL report breaks down the variation in the Measurement System into specific sources. The bar chart shown was created using Excel s Clustered Column Bar Chart to graphically display the Components of Variation. Each cluster of bars represents a source of variation. 33

34 SigmaXL Graphic Output Cheat Sheet SigmaXL provides an R Chart and Xbar Chart by Operator. The R chart consists of the following: - The plotted points are the difference between the largest and smallest measurements on each part for each operator. If the measurements are the same then the range = 0. - The Center Line is the grand average for the process. - The Control Limits represent the amount of variation expected for the subgroup ranges. These limits are calculated using the variation within subgroups. 34

35 SigmaXL Graphic Output Cheat Sheet SigmaXL provides an R Chart and Xbar Chart by Operator. The Xbar Chart compares the part-to-part variation to repeatability. The Xbar chart consists of the following: - The plotted points are the average measurement on each part for each operator. - The Center Line is the overall average for all part measurements by all operators. - The Control Limits (UCL and LCL) are based on the variability between parts and the number of measurements in each average. 35

36 SigmaXL s Gage R&R Multi-Vari Output The Multi-Vari Charts show each Part as a separate graph. Each Operator s response readings are denoted as a vertical line with the top tick corresponding to the Maximum value, bottom tick is the Minimum and the middle tick is the Mean. The horizontal line across each graph is the overall average for each part. 36

37 37 Creating an Interaction Plot

38 SigmaXL Graphic Output Cheat Sheet Pattern: Means: Lines are virtually identical One line is consistently higher or lower than the others Lines are not parallel or they cross Operators are measuring the parts the same That operator is measuring parts consistently higher or lower than the others The operators ability to measure a part depends on which part is being measured (an interaction between operator and part) 38

39 Creating a Multi-Vari Chart to Show All Parts 39

40 SigmaXL Graphic Output Cheat Sheet The By Part Multi-Vari Chart allows us to analyze all of the measurements taken in the study arranged by part. The measurements are represented by dots; the Means by the middle bar. The red line connects the average measurements for each part. 40

41 Creating a Multi-Vari Chart to show all Operators 41

42 SigmaXL Graphic Output Cheat Sheet If the red line is Then Parallel to the x-axis The operators are measuring the parts similarly Not parallel to the x-axis The operators are measuring the parts differently 42

43 Practical Conclusions The Variation due to the measurement system as a percent of study Variation is causing 92.21% of the Variation seen in the process. By AIAG Standards this gage should not be used. By all standards the data being produced by this gage is not valid for analysis. % Tolerance or % Study Variance % Contribution System is 10% or less 1% or less Ideal 10% - 20% 1% - 4% Acceptable 20% - 30% 5% - 9% Marginal 30% or greater 10% or greater Poor 43

44 Repeatability Problems: Calibrate or replace gage. If only occurring with one operator, re-train. Repeatability and Reproducibility Problems Reproducibility Problems: Measurement machines Similar machines Ensure all have been calibrated and the standard measurement method is being utilized. Dissimilar machines One machine is superior. Operators Training and skill level of the operators must be assessed. Operators should be observed to ensure standard procedures are followed. Operator/machine by part interactions Understand why the operator/machine had problems measuring some parts and not others. Re-measure the problem parts Problem could be a result of gage linearity Problem could be fixture problem Problem could be poor gage design 44

45 Design Types Crossed Design A Crossed Design is used only in non-destructive testing and assumes all the parts can be measured multiple times by either operators or multiple machines. Gives the ability to separate part-to-part Variation from measurement system Variation. Assesses Repeatability and Reproducibility. Assesses the interaction between the operator and the part. Nested Design A Nested Design is used for destructive testing and also situations where it is not possible to have all operators or machines measure all the parts multiple times. Destructive testing assumes all the parts within a single batch are identical enough to claim they are the same. Nested designs are used to test measurement systems where it is not possible (or desirable) to send operators with parts to different locations. Do not include all possible combinations of factors. Uses slightly different mathematical model than the Crossed Design. 45

46 Gage R & R Study Gage R&R Study Is a set of trials conducted to assess the Repeatability and Reproducibility of the measurement system Multiple people measure the same characteristic of the same set of multiple units multiple times (a crossed study) Example: 10 units are measured by three people. These units are then randomized and a second measure on each unit is taken A Blind Study is extremely desirable. Best scenario: operator does not know the measurement is a part of a test At minimum: operators should not know which of the test parts they are currently measuring NO, not that kind of R&R! 46

47 Variable Gage R & R Steps Step 1: Call a team meeting to introduce the concepts of the Gage R&R Step 2: Select parts for the study across the range of interest If the intent is to evaluate the measurement system throughout the process range select parts throughout the range If only a small improvement is being made to the process the range of interest is now the improvement range Step 3: Identify the inspectors or equipment you plan to use for the analysis In the case of inspectors explain the purpose of the analysis and that the inspection system is being evaluated not the people Step 4: Calibrate the gage or gages for the study Remember Linearity, Stability and Bias Step 5: Have the first inspector measure all the samples once in random order Step 6: Have the second inspector measure all the samples in random order Continue this process until all the operators have measured all the parts one time This completes the first replicate Step 7: Repeat steps 5 and 6 for the required number of replicates Ensure there is always a delay between the first and second inspection Step 8: Enter the data into SigmaXL to analyze your results Step 9: Draw conclusions to make changes if necessary 47

48 Gage R & R Study Part Allocation From Any Population 10 x 3 x 2 Crossed Design is shown A minimum of two measurements/part/operator is required. Three is better! P a r t s Operator 1 Operator 2 Operator 3 Trial 1 Trial 2 Trial 1 Trial 2 Trial 1 Trial 2 48

49 Data Collection Sheet Create a data collection sheet for: 10 parts 3 operators 2 trials 49

50 50 The Data Collection Sheet

51 Gage R & R Worksheet Gage AIAG2 - SigmaXL Format. 51

52 52 Gage R&R

53 53 Gage R&R Results

54 Graphical Output Use the data from the session output window to recreate the chart below using the following steps: 1. Copy the % Total Variation (TV) column. 2. Paste the column to the right of the % Contribution of Variance Component column. 3. Copy the % Tolerence column. 4. Paste the column to the right of the % Total Variation (TV) column. 5. Highlight the entire table which the % Total Variation (TV) was added. 6. From Excel, Insert> (Chart) Column>2-D Clustered Column. 7. Delete the Variance Component column from this chart. 54

55 Graphical Output The same concept applies to the Response by Operator chart. If there is extreme Variation within operators, then the training of the operators is suspect. Operator Error 55

56 Session Window I can see clearly now! 56

57 Session Window If the Variation due to Gage R & R is high consider ~ Procedures revision? Gage update? 20 % < % Tol GRR < 30% Gage Unacceptable Operator issue? 10 % < % Tol GRR < 20 % Gage Acceptable Tolerance validation? 1 % < % Tol GRR < 10 % Gage Preferable 57

58 Signal Averaging Signal Averaging can be used to reduce Repeatability error when a better gage is not available. Uses average of repeat measurements. Uses Central Limit Theorem to estimate how many repeat measures are necessary. Signal Averaging is a method to reduce Repeatability error in a poor gage when a better gage is not available or when a better gage is not possible. 58

59 Signal Averaging Example Suppose SV/Tolerance is 35%. SV/Tolerance must be 15% or less to use gage. Suppose the Standard Deviation for one part measured by one person many times is 9.5. Determine what the new reduced Standard Deviation should be. 59

60 Signal Averaging Example Determine sample size ~ Using the average of 6 repeated measures will reduce the Repeatability component of measurement error to the desired 15% level. This method should be considered temporary! 60

61 Paper Cutting Exercise Exercise objective: Perform and Analyze a variable MSA Study. 1. Cut a piece of paper into 12 different lengths all fairly close to one another but not too uniform. Label the back of the piece of paper to designate its part number. 2. Perform a variable Gage R&R study as outlined in this module. Use the following guidelines: Number of parts: 12 Number of inspectors: 3 Number of trials: 5 3. Create a SigmaXL data sheet to enter the data into as each inspector performs a length measurement. If possible assign one person to data collection. 4. Analyze the results and discuss with your mentor. 61

62 Attribute MSA A methodology used to assess Attribute Measurement Systems. Attribute Gage Error Repeatability Reproducibility Accuracy They are used in situations where a continuous measure cannot be obtained. It requires a minimum of 5 times as many samples as a continuous study. Disagreements should be used to clarify operational definitions for the categories. Attribute data are usually the result of human judgment (which category does this item belong in). When categorizing items (good/bad; type of call; reason for leaving) you need a high degree of agreement on which way an item should be categorized. 62

63 Attribute MSA Purpose The purpose of an Attribute MSA is: To determine if all inspectors use the same criteria to determine pass from fail. To assess your inspection standards against your customer s requirements. To determine how well inspectors are conforming to themselves. To identify how inspectors are conforming to a known master that includes: How often operators ship defective product How often operators dispose of acceptable product Discover areas where: Training is required Procedures must be developed Standards are not available An Attribute MSA is similar in many ways to the continuous MSA, including the purposes. Do you have any visual inspections in your processes? In your experience how effective have they been? 63

64 Visual Inspection Test Take 60 seconds to count the number of times F appears in this paragraph? The Necessity of Training Farm Hands for First Class Farms in the Fatherly Handling of Farm Live Stock is Foremost in the Eyes of Farm Owners. Since the Forefathers of the Farm Owners Trained the Farm Hands for First Class Farms in the Fatherly Handling of Farm Live Stock, the Farm Owners Feel they should carry on with the Family Tradition of Training Farm Hands of First Class Farmers in the Fatherly Handling of Farm Live Stock Because they Believe it is the Basis of Good Fundamental Farm Management. 64

65 How can we Improve Visual Inspection? Visual Inspection can be improved by: Operator Training & Certification Develop Visual Aids/Boundary Samples Establish Standards Establish Set-Up Procedures Establish Evaluation Procedures Evaluation of the same location on each part. Each evaluation performed under the same lighting. Ensure all evaluations are made with the same standard. Look closely now! 65

66 Attribute MSA (Binary) ~ 1. Open the worksheet Attribute MSA AIAG. This is an example from the AIAG MSA Reference Manual, 3rd Edition, page Click SigmaXL > Measurement Systems Analysis > Attribute MSA (Binary). Ensure that the entire data table is selected. Click Next. 3. Select Part, Appraiser, Assessed Result and Reference as shown. 4. Click OK. The results are shown on the next slide. Attribute Agreement Analysis Attribute MSA is also known as Attribute Agreement Analysis. The response must be binary (e.g. Pass/Fail, Good/Bad, G/NG, Yes/No). 66

67 Attribute MSA (Binary) ~ 67

68 M&M Exercise Exercise objective: Perform and analyze an Attribute MSA Study. You will need the following to complete the study: A bag of M&Ms containing 50 or more pieces. The attribute value for each piece. Three or more inspectors. Judge each M&M as pass or fail. Number 1 Part M&M Attribute Pass The customer has indicated they want a bright, shiny, uncracked M&M. 2 M&M Fail Pick 30 M&Ms out of a package. 3 M&M Pass Enter results into either the Excel template or SigmaXL to draw conclusions. The instructor will represent the customer for the Attribute score. 68

69 Summary At this point you should be able to: Understand Precision & Accuracy Understand Bias, Linearity and Stability Understand Repeatability & Reproducibility Understand the impact of poor gage capability on product quality Identify the various components of Variation Perform the step by step methodology in Variable and Attribute MSA s 69

70 IASSC Certified Lean Six Sigma Black Belt (ICBB) The International Association for Six Sigma Certification (IASSC) is a Professional Association dedicated to growing and enhancing the standards within the Lean Six Sigma Community. IASSC is the only independent third-party certification body within the Lean Six Sigma Industry that does not provide training, mentoring and coaching or consulting services. IASSC exclusively facilitates and delivers centralized universal Lean Six Sigma Certification Standards testing and organizational Accreditations. The IASSC Certified Lean Six Sigma Black Belt (ICBB) is an internationally recognized professional who is well versed in the Lean Six Sigma Methodology. The Certified Black Belt Exam, is a 4 hour 150 question proctored exam. Learn about IASSC Certifications and Exam options at