Welcome to the course, Evaluating the Measurement System. The Measurement System is all the elements that make up the use of a particular gage. Parts, people, the environment, and the gage itself are all part of the measurement system. It is important to determine the uncertainty of the measurement system before using it to evaluate manufacturing processes, development work or experimental results.
Meet Sean. He loves to live the good life. His motto is: Work hard and play hard. He has pretty much lived the theme of this motto since the age of 15. Lately Sean has not been feeling so well so he decided to pay a visit to his doctor. Sean's doctor indicates that he has high blood pressure. Reading Sean the riot act, he indicates that Sean must lower his blood pressure substantially or risk the loss of otherwise excellent health. Having been well programmed by the pharmaceutical industry, the doctor immediately writes Sean a prescription. He asks Sean to come back in 4 weeks.
Initially Sean's reaction is to deny reality. He thinks that the test must have been done on a "bad day". Upon reflection, he decides to attempt to verify the blood pressure results taken at the doctor's office. That evening, Sean decides to drive to the local pharmacy and purchase an off the shelf "home use" blood pressure monitoring device called the Extreme. Upon returning home, Sean takes his blood pressure every 30 minutes for the next 4 hours. He finds that his blood pressure is indeed above normal but that the values seem to vary wildly. Sean now wonders if his blood pressure is truly jumping all over the place or if the meter is reading incorrectly and inconsistently. If it is Sean, he needs to make some serious life style changes...and quickly. If it is the meter, perhaps he can postpone making the changes that will be painful and no fun.
Being a bit of an engineer, Sean decides to first check to see if the monitor is reading the exact value it should be reading each time. A search of the Internet shows an incredible device that mimics the human circulatory system. The device is named Arnold. It is certified to be directly traceable to the National Institute of Standards and Technology (NIST) and can be adjusted to simulate various human blood pressures. Sean first decides to set Arnold at 120/80 and begin taking measurements following the protocol provided in the Extreme. Initially he is only interested in the systolic pressure (the first number). The readings are shown here. The variation in the readings (which
range from 115 to 124) in this scenario is referred to as the estimate of repeatability of the instrument. Repeatability is the variation in measurements obtained with one measurement instrument when used multiple times by one operator. The average of these 5 readings is 121. A graph of these values is shown here. The difference between the standard (120) and the average of our five readings (121) is referred to as the bias of the instrument. Sean now wonders what would happen if another person were to take the five blood pressure readings. He takes 5 readings and another person, Dana, takes 5 readings. The average of Sean's readings is 121 and the average of Dana's readings is 124. The variation in the averages 124 vs 121 is referred to as the reproducibility of the measurement system.
Sean now wants to make sure that the Extreme gives the same reading regardless of the time of day. He turns on the Extreme measurement system at 8:00am and takes 5 readings. He leaves the system on all day and takes 5 additional readings at 4:00pm. The 8:00 am values ranged from 118 to 124. The 4:00 pm values ranged from 119 to 124. Stability(drift) is a measure of how the system performs over time. It is the total variation obtained with a particular device, on the same part, when measuring a single characteristic over time.
Finally, Sean wants to know if the bias changes depending on the blood pressure level. He sets Arnold so the blood pressure is 115, 130 and 145. He then takes 5 readings at each level. The average reading at each level is 115, 133 and 149. The bias values change from 0 at 115 to 4 at 145. Linearity is the difference in the bias values through the expected operating range of the gage. What is the bottomline here. If Sean's blood pressure were actually normal but the measurement device said it was high, Sean's doctor would be wrong to put him on medication. If Sean's blood pressure were actually high but the measurement device said it was normal, Sean could die if not put on an appropriate medication. So, getting the measurement right is a big deal.
Let's switch gears a bit and look at the aspects of measurement a bit more formally. Measurement data is used in many more ways today than in the past. Some examples of how measurement data is used are: to monitor manufacturing processes to monitor product quality to determine what corrective actions to take to adjust manufacturing processes to determine if relationships exist between variables to model new systems
The quality of the data from the measurement devices is critical to good decision making. This is the reason it is imperative to evaluate your measurement system and perform Gage Studies. Bias (sometimes referred to as accuracy) is the difference between the master value and the average of the observed measured values. It refers to the location of the data relative to a master value.
Repeatability is the variation in measurements obtained with one measurement instrument, one appraiser or operator, one characteristic and one part. Reproducibility is as above but with more than one operator. Repeatability and reproducibility are both aspects of the measurement system variation, not the part variation. They test the variance in the tool itself and the variance in the operators of the tool. Gage variance can have significant impacts on the quality of decision making. Suppose you measure your product and it has a lot of variance from part to part.
If the measurement system accounts for only a small portion of that variance, the manufacturing process or the product must be improved. If the measurement system accounts for a large portion of that variance, then the measurement system must be improved. Imagine you are in a quiet room having a conversation with someone. It would be easy to hear their message or "signal". Now try having the same conversation at a ball game or in the subway. It is harder to hear the signal when there is a lot of noise. You may find that you can't get the message even by screaming. When a measurement system has a lot of variation or noise, it is harder to detect the true signal being sent by your process. Conducting a gage study is one way to determine how much noise is present because of your measurement system.
One popular and simple way to quickly assess the repeatability and reproducibility of a measurement system is as follows.
In this example we are going to measure the weight of a part on a platform balance. Five parts will be measured. Parts were selected to cover our range of interest (0.4 to 1.0 ounces). Two operators will measure each part two times. The results are shown here.
Here is the R chart and the Xbar chart for our example. Notice they are separated into two parts each. The first 5 points are from Operator 1 and the second 5 points are from Operator 2. Let's first look at the Range Chart. It shows all points below the upper control limit suggesting variability is consistent over the study. This is good. We do not want to see any point beyond the UCL on a study such as this. A good measurement instrument should have a low centerline on the R chart. The centerline for our example is low and there are no indications of instability. The R Chart indicates that we have an acceptable measurement system.
Now look at the X-bar chart. Your first reaction might be, "Oh, no. Most of the points are out of control." This is actually a good thing during an evaluation of a measurement system. The numerous points above and below the UCL and LCL on the X-bar chart indicates that the measurement system has good discriminating power. Remember that 5 parts were selected over our range of interest (0.4 to 1.0 ounces). The X-bar chart should display numerous points beyond the control limits with the control limits being relatively narrowly distributed about the grand average (X double bar). If the points stayed in control, it would mean that the measurement device could not see a difference between the various parts. Also notice that the pattern for the first five points for Operator 1 is similar to the pattern of the second five points for operator 2. This is also a good sign. It suggests that the two operators obtained similar values when measuring the 5 parts.
Here is the output from Minitab. The table lists numbers for the Repeatability and Reproducibility. Remember, repeatability is the variation in measurements obtained when used multiple times by ONE operator. Reproducibility is the variation in the average measurements obtained when used multiple times by more than one operator. You will also notice that there is a Total Gage R&R Standard Deviation number listed. In this example, it is 0.064318. This is an important metric and will be further discussed on the next page.
One useful way of thinking about the standard deviation of repeatability and reproducibility is to look at the normal distribution. Suppose the true value of the response is 0.8 ounces. The standard deviation is 0.064318. We would expect about 99.73% of our readings to fall within the range of 0.8 +/- (3 x 0.064) or 0.8 +/- 0.19. This would provide values from 0.61 to 0.99. This means that if the true weight of a part is 0.8, it would not be uncommon to get a weight from 0.61 to 0.99 using your measurement system. Would this be a concern to you? Suppose you are using the platform balance as an in process check and the specification for the weight is 0.75 to 0.85. This suggests that even if a part is at the nominal value, it is possible to measure it as below specification as well as above specification based upon the properties of a normal distribution. On the other hand, if the specification is.3 to 1.3, there is a much greater chance a good part will be called good and a bad part will be called bad. Both the Xbar chart and the R chart indicated that our measurement system was capable. Using this measurement device, two operators were able to weigh the parts consistently. No corrective actions are necessary.
The Xbar and R approach can go a long way in allowing you to quickly determine if you have a problem with a gage or not. The advantage to the Xbar and R approach is that it is quick. The disadvantage is that it won't give information about the interaction between the part and the operator. If you suspect such an interaction and have the time, mathematical techniques such as Analysis of Variance (or ANOVA) can be applied. Here is an example of an ANOVA output. The following pages will describe what all these values mean.
ANOVA stands for Analysis of Variance. If you have had courses in Statistics, you may be familiar with the mathematical details of this analysis technique. If not, you will be happy to know that modern software packages such as Minitab will crunch the numbers for you. Repeatability is the ability of the instrument to repeat results if the same sample is analyzed multiple times by the same operator. Reproducibility is the ability of the instrument and multiple operators to produce the same result if the same sample is measured multiple times. Let's discuss Measurement System Error. In this context, error doesn't mean mistakes, it means variation. Measurement System Error can be broken down into two parts - Bias and Precision. There are several terms associated with Bias. Stability: Otherwise known as drift is the total variation in the measurements obtained with a measurement system on the same master or parts when measuring a single characteristic over an extended time. Linearity is the difference in the bias values through the expected operating range of the instrument. Does the bias value remain essentially the same over the range of operation? Resolution: Otherwise known as Discrimination or depth of measurement detail. Can the instrument measure to the depth required for the application? Microns, angstroms, etc. Most calibration labs direct their efforts towards eliminating bias but do little towards improving precision. Gage R&R focuses on the precision of the measurement system - both the Instrument and the operator.
Here is another way to think about it. The total observed variation equals the true process variation plus the variation due to the measurement system. The variation due to the measurement system equals the variation due to repeatability plus the variation due to reproducibility. Remember, repeatability deals with the device and standards. Reproducibility deals with the people and method.
Here are the steps for this more mathematical approach. You will notice that the steps are different than the steps for the simple approach.
Let's look at an example of a Gage R&R Study using the steps discussed in the previous pages. In our example, several operators use a particular gage. We would like to evaluate the following: an effect due to the operator an effect due to the particular part being measured an operator by part interaction effect the effect due to the repeatability of the gage.
The data was entered into a Minitab worksheet for analysis. Notice there is a column for Part number, a column for Operator and a column for the measurement obtained. Select Stat > Quality Tools > Gage Studies(crossed) Let's take a minute to discuss Crossed studies vs Nested Studies. Crossed studies are the type where any setting can be used with any factor. Nested studies are used when certain settings can be used only on certain factors. For example, suppose we are testing two types of popcorn. One is cooked on the burner of a stove and the other is cooked in a microwave. Our response is the number of unpopped kernels. One
factor is time. Could we use the same levels of time for both types of popcorn? No. We would use much shorter times for the microwave popcorn than we would for the regular popcorn. Suppose we use 2-3 minutes for the microwave popcorn and 7-8 minutes for the regular popcorn. A nested design allows us to "nest" the appropriate time within the type of popcorn and still be able to evaluate it correctly. NOTE: In a destructive measurement study, each part can be used only once. Therefore we make the assumption that the parts are close enough to be considered identical. However, we analyze it with a nested design in case this assumption is not perfectly true. In the Gage R&R Study (Crossed) screen enter Part numbers, Operators, and Measurement data. NOTE: These will point to the appropriate columns from your worksheet. Check ANOVA in the Method of Analysis section. Click on the Options button.
Enter the Study variation. This is the number of standard deviations needed to capture 99.73% of your process measurements. For example, +/- 3 standard deviations will capture 99.73% for a normal distribution. Enter the Process tolerance. In our example the specification is -2 to +2. We therefore enter a 4 for the process tolerance. Enter the alpha value to remove interaction terms. We will use the default of 0.25. Click on OK. The Gage R&R Study(crossed) screen will appear again. Click on OK. Minitab will now generate the ANOVA and graphs.
Here is the ANOVA. Look at the P values. The P value for the Parts is 0.000. This is less than 0.05 and indicates that the parts were indeed different. This is good. The P value for the Operators is 0.000. This is also less than 0.05. Unfortunately, this indicates that the operators were different. This is bad. In a Gage R&R study we would like there to be no difference between the operators. Since the operators were different, one or more operators may need to have additional training. The P value for the Part/Operator interaction is 0.974. This is larger than 0.05 and indicates that the interaction is not significant. This is good. Here is some additional information from Minitab. Notice that the % study variation for the Total Gage R&R is 27.86%. The table below gives some general guidelines for evaluating your study. 27.86% falls in the "Measurement System Marginal" category. We have marked it with an X. The % Tolerance is 45.36%. This falls in the "Inadequate" category and has been noted with an X. The number of distinct categories refers to the instruments ability to distinguish one part from another. In general, the more distinct categories, the better. In our example, the number of distinct categories is 4. This is marginal. Unfortunately, our measurement system is inadequate.
Here are the graphs from Minitab. When looking at control charts for gage studies, it is important to know that we want to see "out of control" conditions on the Xbar chart. This indicates that the gage is able to see the difference between the individual parts but that it will measure the same part with very little variation each time. Notice the R Chart by Operator. There was more variability in the readings for Operator B. Perhaps more operator training is necessary. What can you do if the measurement system isn't good enough?
If reproducibility between operators is not good, it is essential to understand the operator differences and to minimize those differences. You could modify the measurement procedure. For example, suppose you are trying to measure the viscosity of a fluid. Some fluids react slowly to shear and it is necessary to allow enough time for full recovery before taking the reading. If you do not do this, the measurements will be inconsistent. If your original procedure called for a 15 second wait time, it may need to be increased to 30 seconds before the reading is taken. You could modify the gage. Using the viscosity example, suppose it is discovered that it is extremely important to take the reading at EXACTLY 15.5 seconds. Obviously, it would be impossible for an operator to take the reading at 15.5 seconds. You may need to automate the procedure by adding a timer and printer directly to the viscometer. This way the viscosity reading is not dependent on a human trying to measure 15.5 seconds. You may need to purchase a new gage. Suppose you need to measure the weight of a feather. Obviously, an ordinary bathroom scale is not going to work for this measurement. What if your budget does not allow for the purchase of a different gage. One possible solution is to have the inspectors take two or more readings and then average the readings. Since this is time consuming, most production managers will not like this solution either, but it will help to reduce measurement error. It would not, however, solve the problem of trying to measure the weight of a feather using a bathroom scale.
Let's talk about Gage R&R studies that require destructive testing. The steps are basically the same as non-destructive testing, however, replicates become a problem. Since the parts are destroyed during the measurement, a "true" replicate reading of the same part cannot be done by each of the operators. Suppose you had two operators testing 10 parts 2 times. 40 parts would be needed for this study.
If you need to use destructive testing, you must be able to assume that all parts within a single batch are identical enough to claim that they are the same part. If you are unable to make that assumption, then part-to-part variation within a batch will mask the measurement system variation. Suppose we would like to perform a Gage R&R study on the measurement system used to measure the pull strength required to open a pop top on a soda can. Every time we open the pop top, that particular soda can will be destroyed. The first step is to select parts for the study across the range of interest. Since we will be destroying the parts, we cannot replicate readings on a given sample. We will create "master' samples that can be sub-divided into smaller, homogeneous samples. We know that different manufacturers of soda produce cans that require different forces to open them. We will use 10 different brands of soda for our study.
Within a brand, we would like the samples to be as homogenous as possible. We make sure that all cans for a given brand are from the same lot number. We then select 4 cans from each brand for our study.
Here is the ANOVA. The P value for the operator represents the variability in measurements across different operators. We want P to be greater than 0.05 so the P value of 8.898 is good. The P value for the part (operator) is 0.000. This represents the variability in measurements across different parts by each operator. We want the P value to be greater than 0.05 so the P value of 0.000 is bad. Here is some more output from Minitab. Based upon our guidelines, this measurement system is marginal.
Notice that one operator appears to be less variable than the other.
Let's look at another example of destructive testing. Suppose you are doing a Gage R&R study on a measurement system that measures tensile strength. Tensile strength is basically the maximum amount of force a part can be subjected to without breaking. In Minitab, select Stat > Quality Tools > Gage Studies > Gage R&R Study (Nested) Here are the results from Minitab. The P value for the operator is 0.996. This is greater than 0.05 and indicates that the operators were the same. This is good.
The % Study Variation was 90.81. This is not good. The % Tolerance was 34.06. This is not good. The number of distinct categories is 1. This is not good. Although the operators appear to be well trained, it looks like the measurement system needs some work. Here are the graphs from Minitab. Let's first look at the R Chart by Operator. Everyone appears to be in control. This is good and indicates that we eliminated as much within sample variation as possible. Let's now look at the Xbar Chart. Since the operators actually measured different samples, we do not expect the same pattern across operators. In this Xbar chart, we do not see the same pattern from operator to operator. This is OK. However, most of the points on the Xbar chart are in control, this is not good. It either indicates that the measurement system does not have enough discriminating power or that we did not select samples from lots that varied enough.
This concludes the course on Evaluating the Measurement System. In this course you have learned how to set up and conduct a Gage R&R study, how to evaluate the results of both crossed and nested variables studies and possible steps to take should a gage not be considered acceptable. Remember, the Measurement System is all the elements that make up the use of a particular gage. Parts, people, the environment, and the gage itself are all part of the measurement system. It is important to determine the uncertainty of the measurement
system before using it to evaluate manufacturing processes. This course has taught you how to do just that.