Four Innovative Methods to Evaluate Attribute Measurement Systems Thomas Rust Reliability Engineer/Trainer Sept. 206 Saving More Lives
Autoliv Global Footprint Japan RoA 0% Europe 7% 3% China 7% Sales 205 35% Americas Sales and technology leader Global Company Sales to all major vehicle manufacturers Minitab Insights 206 Four Methods for Attribute MSA - 2
Autoliv Products Inflatable curtain airbag Vision system Blind spot radar Electronic control unit Driver airbag Steering wheel Side airbag Seatbelt systems Passenger airbag Night driving assist Driver assist radar Minitab Insights 206 Four Methods for Attribute MSA - 3
Our Guiding Principles Minitab Insights 206 Four Methods for Attribute MSA - 4
Our Strategy to Stay Ahead Relentless focus on Operational Excellence Zero Defects by flawless execution One Product One Process to improve cost effectiveness and robustness Innovation to lead industry in Real Life Safety Minitab Insights 206 Four Methods for Attribute MSA - 5
Measurements Minitab Insights 206 Four Methods for Attribute MSA - 6
The Gray Areas Area I Parts are measured bad Area II Parts are sometimes measured good and sometimes bad Due to random variation in the measuring system Area III Parts are measured good LSL I II III II I Target USL Minitab Insights 206 Four Methods for Attribute MSA - 7
AIAG Attribute MSA Methods Cross-Tab Method Kappa Confidence Intervals on Percent Agreement Signal Detection Approach Analytic Method Minitab Insights 206 Four Methods for Attribute MSA - 8
The Attribute Paradigm All we can do is count the number of agreements / disagreements All attribute data are the same It always takes a lot of samples to evaluate and attribute measurement system Minitab Insights 206 Four Methods for Attribute MSA - 9
Breaking The Attribute Paradigm Think There are four types of Attribute Measurement systems. Underlying variable measurement 2. Attribute measurement of a variable product 3. Variable measurement of an attribute product 4. Attribute measurement of a attribute product Minitab Insights 206 Four Methods for Attribute MSA - 0
- Underlying Variable Measurement The characteristic is measured with variable data (or could be), but only the binary result is reported Examples Measuring height and classifying as good or bad Measuring force but only recording if above or below a limit Operator could classify how good/bad but only classifies as pass/fail 92 Minitab Insights 206 Four Methods for Attribute MSA -
Underlying Variable Measurement Method Method Record the variable measurement Treat the same as other variable measurements Gage R&R Have operators rate the how good or bad each part is even if they don t normally A rating system (-0) can be established = Best 4 = Barely passes 5 = Barely fails 0 = Worst Minitab Insights 206 Four Methods for Attribute MSA - 2
Sprocket Inspection All examples and data in this presentation are plausible fabrications A sprocket is supplied to a company to be included in an assembly The end customer will interface with the sprocket so the appearance is important as well as the dimensions The acceptance criteria is based on: Color Surface Roughness Paint Quality (no chipping) Parts are inspected as pass or fail at the supplier, a sorting company, and the assembly company Minitab Insights 206 Four Methods for Attribute MSA - 3
Sprocket Inspection Set-Up and Method 43 Parts were selected Both good and bad parts are found for different criteria Acceptable Color (AC) 9 Bad Color (BC) 3 High Roughness (RO) 7 Chipped Paint (CP) 9 Good Parts (GP) 5 Three judges were selected as the experts to identify the standard rating of each part One from the assembly company (Customer) One from the sorting company One from the supplier Each Part was inspected three times by each judge Parts were presented in a random order Part identity was hidden from each judge Minitab Insights 206 Four Methods for Attribute MSA - 4
Initial Study - Roughness Study Var %Study Var Source StdDev (SD) (6 SD) (%SV) Total Gage R&R 2.386 2.683 95.30 Repeatability.794 0.276 77.8 Reproducibility.24003 7.4402 55.90 Judge.24003 7.4402 55.90 Part-To-Part 0.67238 4.0343 30.3 Total Variation 2.2822 3.3093 00.00 Minitab Insights 206 Four Methods for Attribute MSA - 5
Second Study - Roughness Study Var %Study Var Source StdDev (SD) (6 SD) (%SV) Total Gage R&R.69965 0.979 84.39 Repeatability.37447 8.2468 68.24 Reproducibility 0.9998 5.9989 49.64 Judge 0.9998 5.9989 49.64 Part-To-Part.08074 6.4845 53.66 Total Variation 2.045 2.0849 00.00 Minitab Insights 206 Four Methods for Attribute MSA - 6
Initial Study - Color Study Var %Study Var Source StdDev (SD) (6 SD) (%SV) Total Gage R&R 2.2438 3.459 76.98 Repeatability.98639.983 68.7 Reproducibility.0428 6.253 35.77 Judge 0.79825 4.7895 27.39 Judge*Part 0.67003 4.0202 22.99 Part-To-Part.85980.588 63.83 Total Variation 2.9389 7.4833 00.00 Minitab Insights 206 Four Methods for Attribute MSA - 7
Second Study - Color Study Var %Study Var Source StdDev (SD) (6 SD) (%SV) Total Gage R&R 2.22694 3.366 83.22 Repeatability 0.86923 5.254 32.48 Reproducibility 2.05029 2.308 76.62 Judge.57253 9.4352 58.77 Judge*Part.3562 7.8937 49.7 Part-To-Part.48366 8.909 55.44 Total Variation 2.6759 6.0555 00.00 Minitab Insights 206 Four Methods for Attribute MSA - 8
Initial Study Chipped Paint Study Var %Study Var Source StdDev (SD) (6 SD) (%SV) Total Gage R&R 2.64770 5.8862 90.70 Repeatability.77333 0.6400 60.74 Reproducibility.9662.7967 67.35 Judge.3206 7.9209 45.22 Judge*Part.45699 8.749 49.9 Part-To-Part.22964 7.3778 42.2 Total Variation 2.993 7.558 00.00 Minitab Insights 206 Four Methods for Attribute MSA - 9
Second Study Chipped Paint Study Var %Study Var Source StdDev (SD) (6 SD) (%SV) Total Gage R&R.48823 8.9294 44.20 Repeatability 0.6992 4.952 20.76 Reproducibility.3375 7.8825 39.02 Judge 0.65389 3.9233 9.42 Judge*Part.3945 6.8367 33.84 Part-To-Part 3.0205 8.230 89.70 Total Variation 3.36724 20.2034 00.00 Minitab Insights 206 Four Methods for Attribute MSA - 20
2- Attribute measurement of a variable product The characteristic could be measured with variable data but the measurement system being used can only classify the results in binary Examples Height Pass/Fail Gage Diameter Go/No Go Gage AIAG s Signal Detection Approach and Analytic Method apply to this type of system Minitab Insights 206 Four Methods for Attribute MSA - 2
P ro b a b ility o f P a s s Logistic Regression with Normit Link Function.0 Parts across the range from consistently good to consistently bad 0.8 Measure multiple times 0.6 0.4 Quantify percent pass 0.2 0.0 Model and S-curve Normit Link Function 49.0 49.5 50.0 50.5 5.0 5.5 52.0 52.5 Diameter Minitab Insights 206 Four Methods for Attribute MSA - 22
Go / No-Go Example A Go / No-Go gage is used to verify that a hole is not too small Parts are created that span the range from bad to good with some very close to the limit Each part was measured carefully with a CMM The parts where evaluated with the Go / No-Go gage Using actual operators Each part was measured 0 times Parts were evaluated in a random order Minitab Insights 206 Four Methods for Attribute MSA - 23
Go / No-Go Example Results Binary Logistic Regression with Normit Link Function Normit Link Function P = Φ( α + βx ) μ = -α / β σ = / β Example Calculations μ = -(-36.7) / 2.674 = 5. σ = / (-36.7) = 0.374 Specification Limits => 50-68mm Bias = 5. 50 =.mm %GRR Tol = 6σ/Tol = (6*0.374)/8 = 2.47% Minitab Insights 206 Four Methods for Attribute MSA - 24
Residual Analysis Minitab Insights 206 Four Methods for Attribute MSA - 25
Go / No-Go Results Minitab Macro Minitab Insights 206 Four Methods for Attribute MSA - 26
AIAG Example Minitab Insights 206 Four Methods for Attribute MSA - 27
Data Gage R&R and Attribute MSA Simulation 0 Interval Plot of SDA Error, Cal Error, Logit Error 95% CI for the Mean 0-0 -20-30 SDA Error Cal Error Logit Error The pooled standard deviation is used to calculate the intervals. Minitab Insights 206 Four Methods for Attribute MSA - 28
3 - Variable Measurement of an Attribute Product Count of Paper Sheets by weight Proximity Sensor verifies a component is present Camera detects a correct configuration Optical Sensor Minitab Insights 206 Four Methods for Attribute MSA - 29
Density Attribute MSA Variable measurement of an attribute product Method Record sensor output for good and bad conditions Voltage, output, etc. Create distribution curves Estimate Alpha and Beta Risks Include Lower Bound of Confidence Interval 0.20 0. 5 0. 0 0.05 Histogram of Missing Part, Good Part Normal 78 Variable Missing Part Good Part Mean StDev N 48.83 3.833 45 85.88 2. 78 00 Measurement System includes the gage variation and the product variation 0.00 40 50 60 70 80 90 Data Minitab Insights 206 Four Methods for Attribute MSA - 30
Part Presence Sensor Example A camera system is used to detect if a washer is present The camera is taught what a good condition is When connected to a computer, the camera reports the percent pixels that match the good condition A threshold (limit) is set to define good from bad at 78% Multiple good and bad conditions are measured and the percentages of pixel match are recorded Bad Good Minitab Insights 206 Four Methods for Attribute MSA - 3
Histograms with Limit Histogram of Missing Part, Normal Good Part 0.20 0. 5 78 Variable Missing Part Good Part Mean StDev N 48.83 3.833 45 85.88 2. 78 00 Density 0. 0 0.05 0.00 40 50 60 70 80 90 Data Minitab Insights 206 Four Methods for Attribute MSA - 32
Good Conditions Capability Analysis A capability analysis in Minitab can estimate the z-score of the threshold (limit) A lower bound of the z- score can also be reported The lower bound takes into account the sample size and the uncertainty in the estimate Non-normal distributions can be used when applicable Minitab Insights 206 Four Methods for Attribute MSA - 33
Density Density Alpha Risk Probability of Failing a Good Part 0.005 0.004 0.003 0.002 0.00 0.000 0.005 0.004 0.003 Distribution Plot - Alpha Risk Lower Bound Normal, Mean=0, StDev= 0.0007888 Distribution Percent Pixel Plot Match - Alpha Risk Normal, Mean=0, StDev= -3.6 Using the Z-Score and a normal distribution plot, the alpha risk can be calculated The upper bound of the alpha risk can be calculated from the lower bound Z-score 0.002 0.00 0.000 0.000473 Percent Pixel Match -3.62 Minitab Insights 206 Four Methods for Attribute MSA - 34
Density Density Missing Part Analysis Beta Risk Distribution Plot - Beta Risk Normal, Mean=0, StDev=.0000E- 2 8.0000E- 3 6.0000E- 3 4.0000E- 3 2.0000E- 3 0.0000E+00 7.6 Percent Pixel Match.3656E-4 Distribution Plot - Beta Risk Lower Bound Normal, Mean=0, StDev=.0000E-09 8.0000E- 0 6.0000E- 0 4.0000E- 0 2.0000E- 0 2.0523E-0 0.0000E+00 6.25 Percent Pixel Match Minitab Insights 206 Four Methods for Attribute MSA - 35
Reporting Methods Risk Alpha Risk = 0.047% Alpha LB = 0.0789% Beta Risk =.37E-2% Beta LB = 2.05E-8% PPM Alpha Risk = 47 PPM Alpha LB = 789 PPM Beta Risk =.37E-8 PPM Beta LB = 2.05E-4 PPM Measurement Capability Ppk Good =.2 LB for Ppk Good =.05 Ppk Bad = 2.54 LB for Ppk Bad = 2.08 Reliability Pass a Good Part 99.985% 3 Nines 99.922% (LB) 3 Nines (LB) Fail a Bad Part ~00% 3 Nines 99.99999998% (LB) 9 Nines (LB) Minitab Insights 206 Four Methods for Attribute MSA - 36
Part Presence Sensor Minitab Macro Att. MSA Summary - Good Part vs. Missing Part 0.20 Gage name: Reported by: Part Presence Sensor John Doe Date of study: 6 Jan 206 78 Variable Pass Fail Mean StDev N 85.88 2. 78 00 48.83 3.833 45 Density 0. 5 0. 0 0.05 0.00 Data are assumed to be normally distributed 40 48 56 64 72 80 88 Percent Pixel Match Classification Probability Estimated Pass a Good 0.99985 Fail a Bad.00000000000 Quantile Safety Factor Pass a Good 4.62737 Fail a Bad -0.845725 95% Lower Bound Pass a Good 0.99922 (3 Nines) Fail a Bad 0.99999999980 (9 Nines) Minitab Insights 206 Four Methods for Attribute MSA - 37
Part Presence Sensor Minitab Macro - Residuals Minitab Insights 206 Four Methods for Attribute MSA - 38
Density Density Density Density Other Examples Att. MSA Summary - Good Part vs. Missing Part Att. MSA Summary - Good Part vs. Missing Part 0.20 Gage name: Reported by: Part Presence Sensor John Doe Date of study: 74 6 Jan 206 Variable Pass Fail Mean StDev N 85.88 2. 78 00 48.83 3.833 45 0.20 Gage name: Reported by: Part Presence Sensor John Doe 80 Date of study: 6 Jan 206 Variable Pass Fail Mean StDev N 86.33 2.376 00 69.42 3.47 45 0. 5 0. 0 0.05 0.00 40 50 60 70 80 90 Pixel Percent Match Classification Probability Estimated Pass a Good 0.99999998 Fail a Bad.000000000 Quantile Safety Factor Pass a Good 6.9279 Fail a Bad -0.729999 95% Lower Bound Pass a Good 0.9999998 (6 Nines) Fail a Bad 0.999999964 (7 Nines) 0. 5 0. 0 0.05 0.00 62.4 67. 2 72. 0 76. 8 8. 6 86. 4 9. 2 Pixel Percent Match Classification Probability Estimated Pass a Good 0.996 Fail a Bad 0.9988 Quantile Safety Factor Pass a Good 3.57400 Fail a Bad -0.7826 95% Lower Bound Pass a Good 0.990 ( Nines) Fail a Bad 0.9930 (2 Nines) Data are assumed to be normally distributed Data are assumed to be normally distributed Att. MSA Summary - Good Part vs. Missing Part Att. MSA Summary - Good Part vs. Missing Part 0. 8 0. 6 0. 4 0. 2 0. 0 0.08 0.06 0.04 0.02 0.00 Gage name: Reported by: Part Presence Sensor John Doe 4 28 42 56 70 84 98 65 Sensor Voltage Date of study: 6 Jan 206 Variable Pass Fail Mean StDev N 92.27 2.583 00 2. 6 3.88 45 Classification Probability Estimated Pass a Good.000000000000 Fail a Bad.000000000000 Quantile Safety Factor Pass a Good 7.4029 Fail a Bad -0.652799 95% Lower Bound Pass a Good.000000000000 (>20 Nines) Fail a Bad.000000000000 (>20 Nines) 0. 4 0. 2 0. 0 0.08 0.06 0.04 0.02 0.00 Gage name: Reported by: Part Presence Sensor John Doe 0 4 28 42 56 70 84 98 Sensor Voltage Date of study: 6 Jan 206 78 Variable Pass Fail Mean StDev N 89.64 3.432 00 42.27 20.42 45 Classification Probability Estimated Pass a Good 0.9997 Fail a Bad 0.960 Quantile Safety Factor Pass a Good 7.5822 Fail a Bad -0.78436 95% Lower Bound Pass a Good 0.9985 (2 Nines) Fail a Bad 0.93 ( Nines) Data are assumed to be normally distributed Data are assumed to be normally distributed Minitab Insights 206 Four Methods for Attribute MSA - 39
Evaluation over Time I-MR Chart of Good Parts Individual Value 60 40 20 5 9 3 7 2 25 29 33 37 4 45 UCL=53.44 _ X=42.27 LCL=3.0 Observation Moving Range 30 20 0 0 UCL=3.73 MR=4.20 LCL=0 5 9 3 7 2 25 29 33 37 4 45 Observation Worksheet: Worksheet 2 Minitab Insights 206 Four Methods for Attribute MSA - 40
Risks and Concerns Non normal Multiple failure modes Like catching Terrorists No variation Difficult to collect data Changes over time Variation in the process vs. measurement system Minitab Insights 206 Four Methods for Attribute MSA - 4
4 - Attribute Measurement of an Attribute Product Judgement of a Coin toss Mechanical detection of a part present Binary response of correct set-up Minitab Insights 206 Four Methods for Attribute MSA - 42
Attribute Measurement of an Attribute Product Method Attribute Agreement Analysis with Lower-Bound Confidence Interval Selecting parts At least 25% of parts will be in all categories? yes Randomly select parts from the entire process Caution needs to be taken on selection of good and bad parts A minimum of 50 total parts with at least 50 observations should be used for a reliable analysis no A significant sample of parts from each category is available? no yes Randomly select at least 0 parts from each category Cautiously create at least 0 parts for each category that will represent the process Minitab Insights 206 Four Methods for Attribute MSA - 43
Percent Percent Percent Examples of Attribute Agreement Analysis Assessment Agreement Date of study: Reported by: Name of product: Misc: John Doe Sprocket Chipped Paint Assessment Agreement Date of study: Reported by: Name of product: Misc: John Doe Sprocket Roughness Within Appraisers Appraiser vs Standard Within Appraisers 95 95.0% CI Percent 95 95.0% CI Percent 00 95.0% CI Percent 90 90 80 85 85 60 80 80 75 75 40 70 70 20 65 A B C Appraiser Worksheet: All Data Attribute Example 65 A B Appraiser C 0 Customer Sorting Co Appraiser Supplier Minitab Insights 206 Four Methods for Attribute MSA - 44
Attribute Agreement Analysis Impact of Parts Range Minitab Insights 206 Four Methods for Attribute MSA - 45
Best Practices Mission Possible The New Approach To Attribute MSA Historical Approach to Attribute MSA Minitab Insights 206 Four Methods for Attribute MSA - 46
Thank You! Every year, Autoliv s products save over 30,000 lives and prevent ten times as many severe injuries Minitab Insights 206 Four Methods for Attribute MSA - 47
References MEASUREMENT SYSTEMS ANALYSIS Reference Manual Fourth Edition AIAG Copyright 200, Chrysler Group LLC, Ford Motor Company, General Motors Corporation EMP III, Evaluating the Measurement Process & Using Imperfect Data Donald J. Wheeler Copyright 2006 SPC Press Minitab Insights 206 Four Methods for Attribute MSA - 48