I Really Would Rather Not Randomize My Experiment!!!

Transcription

1 I Really Would Rather Not Randomize My Experiment!!! *Presentation is posted at There are many attendees today! To avoid disrupting the Voice over Internet Protocol (VoIP) system, I will mute all. Please use the Questions feature on GotoWebinar. Feel free to questions to webinar@statease.com which we will answer off-line. -- Wayne By Wayne F. Adams, MS, Applied Stats. Stat-Ease, Inc., Minneapolis, MN wayne@statease.com

2 I Really Would Rather Not Randomize My Experiment!!! If you want to understand the effect of a factor on a response, the factor must change during the experiment. 1 Often, it is not only most convenient but also most effective to run industrial experiments without randomizing all the factors An altered form of the George Box quote, To find out what happens when you change something, it is necessary to change it. 2. The opening quote from George Box s, (1996) Quality Quandaries write up in Quality Engineering 8(3), June 2014 Webinar 2

3 I Really Would Rather Not Randomize My Experiment!!! Temperature in an industrial furnace is a common hard-to-change factor. It costs substantial time to heat or cool the furnace for the next temperature treatment. Potentially changing and/or resetting the temperature every run is impractical. A furnace is better suited to baking a whole batch. June 2014 Webinar 3

4 I Really Would Rather Not Randomize My Experiment!!! Design Setup: The hard-to-change factor treatment combinations are collected into and remain fixed within a group. If at all practical, randomize the order of the groups. The easy-to-change factors are randomized within each group as runs. Understanding this grouping structure is the key to proper analysis. June 2014 Webinar 4

5 I Really Would Rather Not Randomize My Experiment!!! In this example the Temperature is set to levels of 360, 370, and 380. Four corrosion inhibiter coatings are to be baked on to steel bars. June 2014 Webinar 5

6 Run the Experiment C1 C2 C3 C4 The coated bars are inserted randomly into the furnace. The temperature in the furnace is set to desired level, and a batch of bars are baked for a fixed amount of time. Don t forget to close the door! June 2014 Webinar 6

7 That s not random! With furnace temperature, it is not cost effective to randomize the groups. A pattern of 360, 370, 380, 380, 370, 360 is adopted to reduce costs. What problems can this create? Lurking variables can bias the analysis! A good reset between EACH group is necessary. June 2014 Webinar 7

8 Restricting Randomization Considerations Not randomizing the groups increases the risk of confounding the groups with a lurking variable. When a new group starts, even factors that are not changing should be reset. If factors are not reset, then there is yet another restriction on the randomization. Use your subject matter knowledge to determine if not resetting a factor is representative of the population. June 2014 Webinar 8

9 Restricting Randomization Why the structure matters Noise is caused by resetting factor levels. Easy-to-change treatments reset every run. Hard-to-change treatments reset when the group changes. There is one source of noise from the easy-tochange reset and a second from the hard-to-change reset. To account for the differences in associated error, separate half-normal plots and ANOVA s are created to handle the dual error nature of these designs. June 2014 Webinar 9

10 What Happens If It Is Analyzed as Random? The hard-to-change factor (temperature) creates most of the noise in this process. It is difficult to get exactly the same temperature profile in the furnace from batch to batch. If this noise is not accounted for by using the correct analysis methods, it conceals the coating effect. The wrong (too many) degrees of freedom are used to test the temperature effects. If the experiment requires restricting the randomization, take the time to set it up and analyze it with the best tool; use a split-plot design. June 2014 Webinar 10

11 I Really Would Rather Not Randomize My Experiment!!! What is a split plot? Split-Plot costs vs. Randomized costs Transport Truck Example of a Split Plot Mark Anderson and Pat Whitcomb (2007), 2 nd edition, DOE Simplified, Productivity Press, chapter 3. June 2014 Webinar 11

12 Split-Plot Designs Origins The split-plot design originated in the field of agriculture. Experimenters applied different treatments from factors such as fertilizer, irrigation, etc. to a plot of land. The same group of treatments is applied to the whole plot. The whole plots are split into subplots. More factors such as seed cultivar, planting depth, etc. were tested within each subplot. A different whole plot is established to receive new treatments. whole plot 1 subplot subplot subplot subplot subplot subplot subplot subplot whole plot 2 subplot subplot subplot subplot subplot subplot subplot subplot June 2014 Webinar 12

13 Problem (hard-to-change factors): Some factors are more difficult and/or expensive to reset than others. In these cases conducting a experiment where factors can be reset every run isn t practical. Solution: Restricting the randomization of hard to change factors makes it a practical experiment, but, creates a need for a splitplot design and analysis. Split-Plot Designs Modern Usage June 2014 Webinar 13

14 Split-Plot Designs Although hard-to-change factors can be randomized, it would make the experiment too expensive and/or time consuming. Standard analysis assumes factor treatment combinations are randomly assigned to the run order. Restricting the randomization of these factors allows the experiment to be conducted, but creates a more difficult analysis problem. June 2014 Webinar 14

15 Split-Plot Designs The reason split-plot designs are used is to get the proper statistical test from the ANOVA. The differences between split-plot and randomized analysis are as follows: Hard-to-change factor effects are detected with much less power. Easy-to-change effects are detected with a little more power. The proportion of the total noise coming from resetting the hard-to-change factors drives how much the power changes from a randomized design. June 2014 Webinar 15

16 I Really Would Rather Not Randomize My Experiment!!! What is a split plot? Split-Plot costs vs. Randomized costs Transport Truck Example of a Split Plot June 2014 Webinar 16

17 Paper Helicopters June 2014 Webinar 17

18 Paper Helicopters Stat-Ease employees collaborated on a project to gather data via a split-plot design. What is hard to change? June 2014 Webinar 18

19 All the folding hurts my fingers Cutting the tail, body, and folding the wings is a time-consuming process. Rather than making a new helicopter for every run, we can reuse a helicopter over a group of runs. Use a split-plot design Perhaps, call this a split-prop design. June 2014 Webinar 19

20 Half the work almost all the benefit 32-run design for 6 factors, 4 of which are hard to change A Full factorial design for 4 factors is 16 runs. The 4 hard-to-change factors are set up in 16 groups defined by a regular, two-level, factorial design. June 2014 Webinar 20

21 The Split Part 16 Helicopters are used twice with different dropping techniques and the paper clip on or off. June 2014 Webinar 21

22 Half the work Twice the Plots subplot effects Design-Expert Software Time avg Shapiro-Wilk test W-value = p-value = a: Paper b: Rotor c: Body Length d: Body Width E: Clip F: Drop Positive Effects Negative Effects Half-Normal % Probability Half-Normal Plot ae F-Drop E-Clip The Drop technique and the paper clip have a sizable and significant effect on the flight time. There is subjectmatter knowledge supporting the paper-by-clip interaction. The heavier paper is less affected by the clip. Standardized Effect June 2014 Webinar 22

23 The length of the rotors determines flight time. Paper is insignificant on its own, but is in the model to support hierarchy with the paper-by-clip interaction. Half the work Twice the Plots whole plot effects Design-Expert Software Time avg Shapiro-Wilk test W-value = p-value = a: Paper b: Rotor c: Body Length d: Body Width E: Clip F: Drop Positive Effects Negative Effects Half-Normal % Probability a-paper Half-Normal Plot Standardized Effect b-rotor June 2014 Webinar 23

24 Split Plot = Split ANOVA The Anova is split into two parts. The hard-to-change factor effects and interactions are tested in the whole plot section. The easy-to-change effects and interactions are tested in the subplot section. Term Error F p-value Source df df Value Prob > F Whole-plot significant a-paper b-rotor < Subplot < significant E-Clip < F-Drop < ae June 2014 Webinar 24

25 And the Winner is Design-Expert Software Factor Coding: Actual Desirability Design Points X1 = E: Clip X2 = a: Paper Actual Factors b: Rotor = R-Long c: Tail Length = T-Long d: Tail Width = Narrow F: Drop = Bottom a1 24# Nav Ultra a2 26.6# CX22 Desirability Of f Interaction a: Paper (Type) Prediction On E: Clip June 2014 Webinar 25

26 Fingers! My Precious Fingers! What if dropping the helicopter is destructive? Not completely destructive, but perhaps succeeding flight times are shorter than previous flights due to wear and tear. Randomization of the easy-to-change factors provides some protection against this bias. Full randomization (one helicopter per run) prevents it from being a problem. June 2014 Webinar 26

27 It really isn t that hard to randomize The split-plot design had 16 unique helicopters; 8 with each paper. A Min Run Characterize design would require 22 unique helicopters; 11 with each paper. Only 6 more helicopters and this can become a completely randomized fractional design. It has more power to detect the helicopter construction factor effects. It has less power to detect the dropping technique effects. It is 10 fewer trips up the ladder. June 2014 Webinar 27

28 Side by Side Power What if the construction effects are small? 16 helicopters (groups), 32 drops (runs) 22 helicopters (runs), 22 drops (runs) June 2014 Webinar 28

29 I Really Would Rather Not Randomize My Experiment!!! What is a split plot? Split-Plot costs vs. Randomized costs Transport Truck Example of a Split Plot Mark Anderson and Pat Whitcomb (2007), 2 nd edition, DOE Simplified, Productivity Press, chapter 3. June 2014 Webinar 29

30 A Plan Vehicle Testing The maker of a wheeled heavy transport vehicle wants to the demonstrate the fuel efficiency gains from their new transmission. Cheaper tires are suggested for use on heavy transport vehicles to reduce over all costs. June 2014 Webinar 30

31 A Better Plan Combine the Experiments Combining the two factors results in 6 (2 x 3) possible combinations. Two-factor designs need to be replicated to improve power and provide a statistical test for the effects. Consider adding a factor to a design instead of replicating a small number of factors. It is known that different payloads will change the effect of both tires and the transmission. Interactions! It is almost like getting something for free. June 2014 Webinar 31

32 The Best Laid Plans often run into budget limitations Two trucks will be used for this experiment. One truck will receive the new transmission, the other will receive a refurbished old transmission. We are really testing Truck 1 with the new transmission versus Truck 2 with the old. This means we are assuming the trucks are the same as each other and representative of the population of trucks. If the trucks are not the same, then what we think is a transmission effect, might be a truck effect. June 2014 Webinar 32

33 How the experiment will run A randomly selected truck will start the test course, followed by the other. The truck will return. The payload is unloaded. The tires are changed. (the same tires might get put back on). The payload is loaded. (the same payload could be reloaded). The truck gets sent out on the next run June 2014 Webinar 33

34 Sometimes the reset isn t practical New helicopters were constructed for every group. Distinct fields were prepared with different fertilizers. With the furnace temps, there is a natural reset. The door opens; a treated batch comes out; a new batch goes in; the temperature dial is turned to Reset and then to the desired temperature. Once the transmission is installed on a truck, it won t be swapped between trucks or even reinstalled on the same truck. June 2014 Webinar 34

35 Representative Sample This is representative of the true process; transmissions do not get reset between missions. This eliminates hard-to-change factor reset noise. THIS IS ONLY A GOOD IDEA WHEN IT IS REPRESENTATIVE OF THE POPULATION! Plan for a reset between groups. Only experiment without resets when absolutely necessary. June 2014 Webinar 35

36 This experiment consists of: Two transmissions (std. and new): Three different priced tires ($750, $1000, and $1500): $750 $1000 $1500 Two payloads: an armored personnel carrier (APC) ~ 20 tons and a Tank (M1) ~ 70 tons. Split-Plot Designs Setting up Std. APC New 2 x 3 x 2 = 12 runs is best fit by a General Factorial design! M1 June 2014 Webinar 36

37 Split-Plot Designs Factors, Replicates and Blocking Four replicates (48 runs) are necessary to create a reasonable chance of detecting a two standard deviation change in the response, if it exists. Because we do not plan to run 24 hours a day, each replicate will take place on its own day. June 2014 Webinar 37

38 Split-Plot Designs Blocking The same 12 runs will be conducted within each day. Does that sound like another restriction on the randomization? Instead of making Day of the Experiment another hard-to-change factor it is blocked. June 2014 Webinar 38

39 Split-Plot Designs Power My rule for determining whether something can be treated as a hard-to-change factor is answering the question, can it be replicated? Can the setting of June 16, 2014 be replicated, once the calendar has turned to June 17? Block day-to-day variation from influencing the analysis. Anything that is changing during an experiment which is not a factor of interest can be blocked, with a few exceptions. June 2014 Webinar 39

40 It is worth saying again, twice Not randomizing increases the risk of confounding a factor change with a lurking variable. Not resetting a factor level from group to group or even run to run can bias the estimate of noise. This in turn biases the statistical tests. Not separating the noise (testing like everything was randomized) with the proper analysis can mislead the model. June 2014 Webinar 40

41 Factorial Split-Plot Workshop New workshop available! Factorial Split-Plot Designs for Hard-to-Change Factors Is now offered as a ½-day training option after select public Experiment Design Made Easy workshops or onsite at your convenience. June 2014 Webinar 41

42 How to get help Search publications posted at In Stat-Ease software press for Screen Tips, view reports in annotated mode, look for context-sensitive Help (right-click) or search the main Help system. Explore Experiment Design Forum and post your question (if not previously answered). for answers from Stat-Ease s staff of statistical consultants. Call and ask for statistical help. June 2014 Webinar 42

43 Support for DOE Quarterly Stat-Teaser newsletter (if you don t opt out) Bi-Monthly DOE FAQ Alert (if you don t opt out) Subscribe at: Website: Past issues of Stat-Teaser and DOE FAQ Alert Dozens of case-study reprints Search capability to find the information you need June 2014 Webinar 43