Four Innovative Methods to Evaluate Attribute Measurement Systems

Transcription

1 Four Innovative Methods to Evaluate Attribute Measurement Systems Thomas Rust Reliability Engineer/Trainer Sept. 206 Saving More Lives

2 Autoliv Global Footprint Japan RoA 0% Europe 7% 3% China 7% Sales % Americas Sales and technology leader Global Company Sales to all major vehicle manufacturers Minitab Insights 206 Four Methods for Attribute MSA - 2

3 Autoliv Products Inflatable curtain airbag Vision system Blind spot radar Electronic control unit Driver airbag Steering wheel Side airbag Seatbelt systems Passenger airbag Night driving assist Driver assist radar Minitab Insights 206 Four Methods for Attribute MSA - 3

4 Our Guiding Principles Minitab Insights 206 Four Methods for Attribute MSA - 4

5 Our Strategy to Stay Ahead Relentless focus on Operational Excellence Zero Defects by flawless execution One Product One Process to improve cost effectiveness and robustness Innovation to lead industry in Real Life Safety Minitab Insights 206 Four Methods for Attribute MSA - 5

6 Measurements Minitab Insights 206 Four Methods for Attribute MSA - 6

7 The Gray Areas Area I Parts are measured bad Area II Parts are sometimes measured good and sometimes bad Due to random variation in the measuring system Area III Parts are measured good LSL I II III II I Target USL Minitab Insights 206 Four Methods for Attribute MSA - 7

8 AIAG Attribute MSA Methods Cross-Tab Method Kappa Confidence Intervals on Percent Agreement Signal Detection Approach Analytic Method Minitab Insights 206 Four Methods for Attribute MSA - 8

9 The Attribute Paradigm All we can do is count the number of agreements / disagreements All attribute data are the same It always takes a lot of samples to evaluate and attribute measurement system Minitab Insights 206 Four Methods for Attribute MSA - 9

10 Breaking The Attribute Paradigm Think There are four types of Attribute Measurement systems. Underlying variable measurement 2. Attribute measurement of a variable product 3. Variable measurement of an attribute product 4. Attribute measurement of a attribute product Minitab Insights 206 Four Methods for Attribute MSA - 0

11 - Underlying Variable Measurement The characteristic is measured with variable data (or could be), but only the binary result is reported Examples Measuring height and classifying as good or bad Measuring force but only recording if above or below a limit Operator could classify how good/bad but only classifies as pass/fail 92 Minitab Insights 206 Four Methods for Attribute MSA -

12 Underlying Variable Measurement Method Method Record the variable measurement Treat the same as other variable measurements Gage R&R Have operators rate the how good or bad each part is even if they don t normally A rating system (-0) can be established = Best 4 = Barely passes 5 = Barely fails 0 = Worst Minitab Insights 206 Four Methods for Attribute MSA - 2

13 Sprocket Inspection All examples and data in this presentation are plausible fabrications A sprocket is supplied to a company to be included in an assembly The end customer will interface with the sprocket so the appearance is important as well as the dimensions The acceptance criteria is based on: Color Surface Roughness Paint Quality (no chipping) Parts are inspected as pass or fail at the supplier, a sorting company, and the assembly company Minitab Insights 206 Four Methods for Attribute MSA - 3

14 Sprocket Inspection Set-Up and Method 43 Parts were selected Both good and bad parts are found for different criteria Acceptable Color (AC) 9 Bad Color (BC) 3 High Roughness (RO) 7 Chipped Paint (CP) 9 Good Parts (GP) 5 Three judges were selected as the experts to identify the standard rating of each part One from the assembly company (Customer) One from the sorting company One from the supplier Each Part was inspected three times by each judge Parts were presented in a random order Part identity was hidden from each judge Minitab Insights 206 Four Methods for Attribute MSA - 4

15 Initial Study - Roughness Study Var %Study Var Source StdDev (SD) (6 SD) (%SV) Total Gage R&R Repeatability Reproducibility Judge Part-To-Part Total Variation Minitab Insights 206 Four Methods for Attribute MSA - 5

16 Second Study - Roughness Study Var %Study Var Source StdDev (SD) (6 SD) (%SV) Total Gage R&R Repeatability Reproducibility Judge Part-To-Part Total Variation Minitab Insights 206 Four Methods for Attribute MSA - 6

17 Initial Study - Color Study Var %Study Var Source StdDev (SD) (6 SD) (%SV) Total Gage R&R Repeatability Reproducibility Judge Judge*Part Part-To-Part Total Variation Minitab Insights 206 Four Methods for Attribute MSA - 7

18 Second Study - Color Study Var %Study Var Source StdDev (SD) (6 SD) (%SV) Total Gage R&R Repeatability Reproducibility Judge Judge*Part Part-To-Part Total Variation Minitab Insights 206 Four Methods for Attribute MSA - 8

19 Initial Study Chipped Paint Study Var %Study Var Source StdDev (SD) (6 SD) (%SV) Total Gage R&R Repeatability Reproducibility Judge Judge*Part Part-To-Part Total Variation Minitab Insights 206 Four Methods for Attribute MSA - 9

20 Second Study Chipped Paint Study Var %Study Var Source StdDev (SD) (6 SD) (%SV) Total Gage R&R Repeatability Reproducibility Judge Judge*Part Part-To-Part Total Variation Minitab Insights 206 Four Methods for Attribute MSA - 20

21 2- Attribute measurement of a variable product The characteristic could be measured with variable data but the measurement system being used can only classify the results in binary Examples Height Pass/Fail Gage Diameter Go/No Go Gage AIAG s Signal Detection Approach and Analytic Method apply to this type of system Minitab Insights 206 Four Methods for Attribute MSA - 2

22 P ro b a b ility o f P a s s Logistic Regression with Normit Link Function.0 Parts across the range from consistently good to consistently bad 0.8 Measure multiple times Quantify percent pass Model and S-curve Normit Link Function Diameter Minitab Insights 206 Four Methods for Attribute MSA - 22

23 Go / No-Go Example A Go / No-Go gage is used to verify that a hole is not too small Parts are created that span the range from bad to good with some very close to the limit Each part was measured carefully with a CMM The parts where evaluated with the Go / No-Go gage Using actual operators Each part was measured 0 times Parts were evaluated in a random order Minitab Insights 206 Four Methods for Attribute MSA - 23

24 Go / No-Go Example Results Binary Logistic Regression with Normit Link Function Normit Link Function P = Φ( α + βx ) μ = -α / β σ = / β Example Calculations μ = -(-36.7) / = 5. σ = / (-36.7) = Specification Limits => 50-68mm Bias = =.mm %GRR Tol = 6σ/Tol = (6*0.374)/8 = 2.47% Minitab Insights 206 Four Methods for Attribute MSA - 24

25 Residual Analysis Minitab Insights 206 Four Methods for Attribute MSA - 25

26 Go / No-Go Results Minitab Macro Minitab Insights 206 Four Methods for Attribute MSA - 26

27 AIAG Example Minitab Insights 206 Four Methods for Attribute MSA - 27

28 Data Gage R&R and Attribute MSA Simulation 0 Interval Plot of SDA Error, Cal Error, Logit Error 95% CI for the Mean SDA Error Cal Error Logit Error The pooled standard deviation is used to calculate the intervals. Minitab Insights 206 Four Methods for Attribute MSA - 28

29 3 - Variable Measurement of an Attribute Product Count of Paper Sheets by weight Proximity Sensor verifies a component is present Camera detects a correct configuration Optical Sensor Minitab Insights 206 Four Methods for Attribute MSA - 29

30 Density Attribute MSA Variable measurement of an attribute product Method Record sensor output for good and bad conditions Voltage, output, etc. Create distribution curves Estimate Alpha and Beta Risks Include Lower Bound of Confidence Interval Histogram of Missing Part, Good Part Normal 78 Variable Missing Part Good Part Mean StDev N Measurement System includes the gage variation and the product variation Data Minitab Insights 206 Four Methods for Attribute MSA - 30

31 Part Presence Sensor Example A camera system is used to detect if a washer is present The camera is taught what a good condition is When connected to a computer, the camera reports the percent pixels that match the good condition A threshold (limit) is set to define good from bad at 78% Multiple good and bad conditions are measured and the percentages of pixel match are recorded Bad Good Minitab Insights 206 Four Methods for Attribute MSA - 3

32 Histograms with Limit Histogram of Missing Part, Normal Good Part Variable Missing Part Good Part Mean StDev N Density Data Minitab Insights 206 Four Methods for Attribute MSA - 32

33 Good Conditions Capability Analysis A capability analysis in Minitab can estimate the z-score of the threshold (limit) A lower bound of the z- score can also be reported The lower bound takes into account the sample size and the uncertainty in the estimate Non-normal distributions can be used when applicable Minitab Insights 206 Four Methods for Attribute MSA - 33

34 Density Density Alpha Risk Probability of Failing a Good Part Distribution Plot - Alpha Risk Lower Bound Normal, Mean=0, StDev= Distribution Percent Pixel Plot Match - Alpha Risk Normal, Mean=0, StDev= -3.6 Using the Z-Score and a normal distribution plot, the alpha risk can be calculated The upper bound of the alpha risk can be calculated from the lower bound Z-score Percent Pixel Match Minitab Insights 206 Four Methods for Attribute MSA - 34

35 Density Density Missing Part Analysis Beta Risk Distribution Plot - Beta Risk Normal, Mean=0, StDev=.0000E E E E E E Percent Pixel Match.3656E-4 Distribution Plot - Beta Risk Lower Bound Normal, Mean=0, StDev=.0000E E E E E E E Percent Pixel Match Minitab Insights 206 Four Methods for Attribute MSA - 35

36 Reporting Methods Risk Alpha Risk = 0.047% Alpha LB = % Beta Risk =.37E-2% Beta LB = 2.05E-8% PPM Alpha Risk = 47 PPM Alpha LB = 789 PPM Beta Risk =.37E-8 PPM Beta LB = 2.05E-4 PPM Measurement Capability Ppk Good =.2 LB for Ppk Good =.05 Ppk Bad = 2.54 LB for Ppk Bad = 2.08 Reliability Pass a Good Part % 3 Nines % (LB) 3 Nines (LB) Fail a Bad Part ~00% 3 Nines % (LB) 9 Nines (LB) Minitab Insights 206 Four Methods for Attribute MSA - 36

37 Part Presence Sensor Minitab Macro Att. MSA Summary - Good Part vs. Missing Part 0.20 Gage name: Reported by: Part Presence Sensor John Doe Date of study: 6 Jan Variable Pass Fail Mean StDev N Density Data are assumed to be normally distributed Percent Pixel Match Classification Probability Estimated Pass a Good Fail a Bad Quantile Safety Factor Pass a Good Fail a Bad % Lower Bound Pass a Good (3 Nines) Fail a Bad (9 Nines) Minitab Insights 206 Four Methods for Attribute MSA - 37

38 Part Presence Sensor Minitab Macro - Residuals Minitab Insights 206 Four Methods for Attribute MSA - 38

39 Density Density Density Density Other Examples Att. MSA Summary - Good Part vs. Missing Part Att. MSA Summary - Good Part vs. Missing Part 0.20 Gage name: Reported by: Part Presence Sensor John Doe Date of study: 74 6 Jan 206 Variable Pass Fail Mean StDev N Gage name: Reported by: Part Presence Sensor John Doe 80 Date of study: 6 Jan 206 Variable Pass Fail Mean StDev N Pixel Percent Match Classification Probability Estimated Pass a Good Fail a Bad Quantile Safety Factor Pass a Good Fail a Bad % Lower Bound Pass a Good (6 Nines) Fail a Bad (7 Nines) Pixel Percent Match Classification Probability Estimated Pass a Good Fail a Bad Quantile Safety Factor Pass a Good Fail a Bad % Lower Bound Pass a Good ( Nines) Fail a Bad (2 Nines) Data are assumed to be normally distributed Data are assumed to be normally distributed Att. MSA Summary - Good Part vs. Missing Part Att. MSA Summary - Good Part vs. Missing Part Gage name: Reported by: Part Presence Sensor John Doe Sensor Voltage Date of study: 6 Jan 206 Variable Pass Fail Mean StDev N Classification Probability Estimated Pass a Good Fail a Bad Quantile Safety Factor Pass a Good Fail a Bad % Lower Bound Pass a Good (>20 Nines) Fail a Bad (>20 Nines) Gage name: Reported by: Part Presence Sensor John Doe Sensor Voltage Date of study: 6 Jan Variable Pass Fail Mean StDev N Classification Probability Estimated Pass a Good Fail a Bad Quantile Safety Factor Pass a Good Fail a Bad % Lower Bound Pass a Good (2 Nines) Fail a Bad 0.93 ( Nines) Data are assumed to be normally distributed Data are assumed to be normally distributed Minitab Insights 206 Four Methods for Attribute MSA - 39

40 Evaluation over Time I-MR Chart of Good Parts Individual Value UCL=53.44 _ X=42.27 LCL=3.0 Observation Moving Range UCL=3.73 MR=4.20 LCL= Observation Worksheet: Worksheet 2 Minitab Insights 206 Four Methods for Attribute MSA - 40

41 Risks and Concerns Non normal Multiple failure modes Like catching Terrorists No variation Difficult to collect data Changes over time Variation in the process vs. measurement system Minitab Insights 206 Four Methods for Attribute MSA - 4

42 4 - Attribute Measurement of an Attribute Product Judgement of a Coin toss Mechanical detection of a part present Binary response of correct set-up Minitab Insights 206 Four Methods for Attribute MSA - 42

43 Attribute Measurement of an Attribute Product Method Attribute Agreement Analysis with Lower-Bound Confidence Interval Selecting parts At least 25% of parts will be in all categories? yes Randomly select parts from the entire process Caution needs to be taken on selection of good and bad parts A minimum of 50 total parts with at least 50 observations should be used for a reliable analysis no A significant sample of parts from each category is available? no yes Randomly select at least 0 parts from each category Cautiously create at least 0 parts for each category that will represent the process Minitab Insights 206 Four Methods for Attribute MSA - 43

44 Percent Percent Percent Examples of Attribute Agreement Analysis Assessment Agreement Date of study: Reported by: Name of product: Misc: John Doe Sprocket Chipped Paint Assessment Agreement Date of study: Reported by: Name of product: Misc: John Doe Sprocket Roughness Within Appraisers Appraiser vs Standard Within Appraisers % CI Percent % CI Percent % CI Percent A B C Appraiser Worksheet: All Data Attribute Example 65 A B Appraiser C 0 Customer Sorting Co Appraiser Supplier Minitab Insights 206 Four Methods for Attribute MSA - 44

45 Attribute Agreement Analysis Impact of Parts Range Minitab Insights 206 Four Methods for Attribute MSA - 45

46 Best Practices Mission Possible The New Approach To Attribute MSA Historical Approach to Attribute MSA Minitab Insights 206 Four Methods for Attribute MSA - 46

47 Thank You! Every year, Autoliv s products save over 30,000 lives and prevent ten times as many severe injuries Minitab Insights 206 Four Methods for Attribute MSA - 47

48 References MEASUREMENT SYSTEMS ANALYSIS Reference Manual Fourth Edition AIAG Copyright 200, Chrysler Group LLC, Ford Motor Company, General Motors Corporation EMP III, Evaluating the Measurement Process & Using Imperfect Data Donald J. Wheeler Copyright 2006 SPC Press Minitab Insights 206 Four Methods for Attribute MSA - 48