A comparison of the performance of statistical quality control charts in a dairy production system through stochastic simulation

Transcription

1 AGRICULTURAL SYSTEMS Agricultural Systems 84 (2005) A comparison of the performance of statistical quality control charts in a dairy production system through stochastic simulation A. de Vries *, B.J. Conlin Department of Animal Science, University of Minnesota, 1364 Eckles Avenue, Saint Paul, MN , USA Received 2 May 2003; received in revised form 18 May 2004; accepted 7 June 2004 Abstract Statistical quality control charts are plots of observations over time applied to a characteristic of a system to distinguish between normal variation, when the system should be left alone, and unexpected true changes, which should be signaled as soon as possible so the cause of the changes can be corrected. In practice, the performance of control charts (measured as the average time to first signal, either when the system has not changed (causing a false alarm) or for a true change of a certain size) is unknown if the variability of the system is not quantified, such as in a dairy production system. Therefore, a Monte Carlo simulation model of a 1000-cow dairy herd was used to compare the performance of various control chart designs for days to first service (DFS) and estrous detection index (EDI) to signal decreases in estrous detection efficiency (EDE). Shewhart and cumulative sum (cusum) control charts were designed for period lengths of seven days. The observed average time between false alarms, measured in days, on Shewhart charts with 3-sigma control limits and three supplementary runs tests was much shorter than expected. In addition, control charts for the variance of DFS (assuming a normal distribution) or the mean of EDI (assuming a binomial distribution) had high false alarm * Corresponding author. Present address: Department of Animal Sciences, University of Florida, Building 459, Shealy Drive, P.O. Box , Gainesville, FL , USA. Tel.: ; fax: address: devries@animal.ufl.edu (A. de Vries) X/$ - see front matter Ó 2004 Elsevier Ltd. All rights reserved. doi: /j.agsy

2 318 A. de Vries, B.J. Conlin / Agricultural Systems 84 (2005) rates. Control limits were then varied such that the observed average time between false alarms on the control charts were similar to the goal of 400 or 800 days. This allowed for a fair comparison to signal true decreases in EDE as soon as possible. Large decreases in EDE were signaled earlier on most control charts than small decreases. Normal cusum control charts applied to the mean of EDI signaled true decreases in EDE the earliest: it took on average 38 days to signal a decrease to 55% EDE, 15 days to 45% EDE, or 10 days to 35% EDE, when the average time between false alarms was 400 days. Control charts designed for lower false alarm rates and those applied to DFS took longer to signal decreases in EDE. The results suggest that stochastic simulation of complex production systems can provide insight in the performance of control charts in practice. Furthermore, a combination of the Shewhart and cusum control charts based on the normal distribution applied to EDI is a good choice when the goal is to signal a decrease in EDE as soon as possible. Ó 2004 Elsevier Ltd. All rights reserved. Keywords: Dairy; Statistical quality control; Estrous detection; Shewhart; Cumulative sum 1. Introduction There is general agreement on the need to monitor herd performance, but it is not known how to do it effectively (Dohoo, 1993). Typically, herd measures are calculated from data collected during a fixed period (e.g. week or month) and compared to the previous period or standards (Reneau and Kinsel, 2001). Statistical analysis of herd performance data is seldom done, leaving doubt about how to interpret the data and, thus, which course of action is best (Noordhuizen et al., 1992). Statistical quality control refers to the use of statistically based methods to monitor, control, evaluate, analyse, and improve processes in production systems (Gaafar and Keats, 1992). The main quality control technique is the control chart, which is a plot of observations over time with the objective to distinguish between normal variability, when the system should be left alone, and special variability, which might indicate a fundamental change or problem. Originally applied in manufacturing, control charts have since been used in many non-manufacturing and service environments (Saniga and Shirland, 1977; Montgomery, 1997). More recently, control charts have been proposed and applied in health care (Benneyan, 1998; Kelley, 1999) and livestock production (Polson et al., 1998; Reneau and Kinsel, 2001). When applied to livestock production, the control charts are used to monitor herd performance. The success of control charts depends on avoiding unnecessary interference due to false alarms while true unplanned changes in the system need to be signaled as early as possible so that interference is justified and corrective action can be taken (Quesenberry, 1997). For example, the success of estrous detection in a dairy production system varies from day to day due to random causes and, occasionally, a real change in the efficiency of estrous detection. False alarms are costly, because each signal needs to be followed up by an in-depth investigation of the root causes of estrous detection success, such as who observes estrus, how much time is spend, and how well do cows show estrous signs. Each investigation costs at least time to conduct.

3 A. de Vries, B.J. Conlin / Agricultural Systems 84 (2005) On the other hand, a real decrease in the success of estrous detection is costly too (Marsh et al., 1987; Plaizier et al., 1997). The longer it takes before the decrease is signaled and can be corrected, the larger the economic loss is. Therefore, both the rate of false alarms and the time needed to signal a real change should be small. The performance of control charts generally refers to their average number of observations (or time) between false alarms (signals when the system has not changed) and the average number of observations (or time) needed to signal true changes, which depend on the size of the true change. Typically, the performance of control charts is derived analytically from known mathematical probability distributions such as the normal or binomial distribution. In addition, simple Monte Carlo simulation has been used when an analytic solution is not feasible (Walker et al., 1991; Quesenberry, 1997; Hawkins and Olwell, 1998). The performance of control charts in a complex production system, such as a dairy herd, cannot be found analytically because most observations have unknown distributions and are not independent. Moreover, there may be a lag between the change in the system and the time when the observations become available (Fetrow et al., 1997). For example, the result of an insemination is not known until a return to estrus or pregnancy diagnosis several weeks after the insemination. Hence, a change in the probability of conception cannot be signaled immediately. Furthermore, a change in reproductive efficiency may affect the number of open and pregnant cows in a herd. This will affect the sample sizes of reproductive measures and therefore the performance of control charts applied to these measures. Therefore, the specific variability and dynamics of a production system affect the performance of control charts applied to that system. The performance of control charts in dairy production systems is largely unknown. The rate of false alarms on standard control charts may be much higher than desired, which could result in considerable frustration, because the root causes of the signal cannot be identified (Caulcutt, 1995). In addition, the lag and momentum of different reproductive measures have not been compared objectively. It is largely unclear how different control chart designs vary in performance in dairy production systems. To gain some insight in the performance of control charts under practical conditions, we used a Monte Carlo simulation model of a dairy production system (de Vries and Conlin, 2003). In that study, we measured the performance of some control chart designs to signal a decrease in estrous detection efficiency (EDE), defined as the proportion of true estruses that are observed. The measure estrous detection ratio was used in that study to monitor the rate of EDE. This stochastic simulation approach allows for a quantitative study of the lag and momentum of various measures that may be used to monitor a dairy production system. Monte Carlo simulation has been used before to study the effects of permanent changes in dairy herd reproduction (Oltenacu et al., 1981; Marsh et al., 1987; Sørensen et al., 1992), but the focus on the simulated variability to estimate the performance of control charts appears to be new. The current study had three objectives. The first objective was to illustrate the use of the Monte Carlo simulation approach to compare the timeliness of days to first

4 320 A. de Vries, B.J. Conlin / Agricultural Systems 84 (2005) service (DFS, the time between calving and first service) and estrous detection index (EDI, the fraction expected estruses that is observed) to signal a decrease in EDE. Both measures are used in practice. Fetrow et al. (1990) recommended EDI as a measure of EDE. Jansen et al. (1987) suggested that both DFS and EDI should be used as management aids to monitor herd fertility. Days to first service is an example of a measure that is calculated from continuous observations while EDI is calculated from binary observations. These data characteristics require different types of control charts. The second objective was to describe the performance of traditional Shewhart control charts and the use of three supplementary runs tests applied to DFS and EDI to signal a decrease in EDE. The traditional Shewhart chart has control limits set at three standard errors from the center line. The use of supplementary runs tests to signal more special patterns in the observations has been advocated (Reneau and Kinsel, 2001), because they may enhance the performance of control charts to signal small decreases in EDE. However, the rate of false alarms may also increase. The third objective was to compare the performance of a wide variety of plausible control chart designs applied to DFS and EDI. The types of control charts studied were the less known cumulative sum (cusum) and non-parametric control charts, among more traditional Shewhart charts. An algorithm was developed to design control charts with similar false alarm rates to allow for fair comparisons of their performances to detect true decreases in EDE. 2. Materials and methods 2.1. General aspects of control charts The first control charts were developed by W.A. Shewhart in the 1920s and are referred to as Shewhart charts. A Shewhart chart is a time-ordered graphical display of observations with a center line and a lower and upper control limit which are set at a certain distance below and above the center line (Fig. 1). The observations may be data directly measured from the system or are statistics calculated from these data, such as the mean, standard deviation, or moving range. The observations are plotted on the control chart as they become available. Control limits help the decision maker to distinguish between observations that are within the control limits and therefore most likely within normal random variability, and those that are not (Montgomery, 1997). Control limits set closer to the center line result in more false alarms, but will signal true changes in the system earlier. Traditional Shewhart charts have control limits set at three standard errors ( 3- sigma ) from the center line, which Shewhart found to be a reasonable economic balance between the false alarm rate and the time needed to signal a true change. Three-sigma control limits are claimed to typically work well in practice, regardless of the distribution of the observations (Wheeler and Chambers, 1992). Because this approach does not consider the probability distribution of the observations, the false alarm rate could be quite different from what was intended. Another approach there-

5 A. de Vries, B.J. Conlin / Agricultural Systems 84 (2005) Estrous detection index Week Fig. 1. Example of a Shewhart control chart for estrous detection index (center line ( ), control limits ()). The control chart signals for the first time in week 15 ( ). fore is to calculate control limits based on the probability that an observation takes on certain unlikely values given the assumed mathematical probability distribution, such as a normal or binomial distribution (Woodall and Montgomery, 1999). A third, intermediate, approach is to assume a mathematical probability distribution but always set both lower and upper control limits equally wide from the center line based on the theoretical standard error. The fact that the mathematical probability distribution may be skewed is ignored. Non-parametric control charts, which do not require that a mathematical probability distribution is assumed, have also been developed (Bakir, 1998). All these approaches to the design of control limits were evaluated in this study. The basic test for all control charts is to signal a likely change when an observation falls outside a control limit (Test 1 in Fig. 2). Supplementary runs tests (in addition to the basic test) have been developed which all aim to signal a special pattern in the observations earlier. Fig. 2 shows three recommended supplementary runs tests in addition to the basic test (Wheeler, 1995; Quesenberry, 1997). For example, a signal is triggered when two out of three observations are more than two standard errors from the center line (Test 3 in Fig. 2). Several authors recommend using cusum charts instead of supplementary runs tests to signal small changes in a system (Montgomery, 1997; Hawkins and Olwell, 1998). A cusum is a running total of the deviations between the observed and mean values (Hawkins and Olwell, 1998; see Appendix A for details). Two separate cusums are calculated; one for an increase in the parameter of interest (an upward cusum) and one for a decrease in the parameter of interest (a downward cusum). Fig. 3 shows a cusum chart with both the upward and downward cusum. Cusum charts are designed with probability control limits. The basic test (Test 1) is the only test used with cusum charts Performance measures of control charts The performance of control charts is generally expressed as the average run length (ARL). The run length is the number of periods from a starting point

6 322 A. de Vries, B.J. Conlin / Agricultural Systems 84 (2005) A B C C B A A B C C B A Test 1: 1 point beyond zone A Test 3: At least 2 out of 3 points in a row in zone A or beyond o A B C C B A A B C C B A Test 2: 9 points in a row on same side of center line Test 4: At least 4 out of 5 points in a row in zone B or beyond Fig. 2. The basic test (Test 1) and three supplementary runs tests (Quesenberry, 1997). The circle ( ) indicates that a signal is triggered on the control chart. Zone C: between the center line and 1 standard error; zone B: between one and two standard errors; zone A, between two standard errors and the control limit (placed at three standard errors from the center line). Cumulative sum of estrous detection index Week Fig. 3. Example of a cusum control chart for estrous detection index (EDI) (center line ( ), control limits ( ), upward cusum (h), downward cusum ( )). The upward cusum monitors an increase in EDI. A separate downward cusum monitors a decrease in EDI. The control chart signals for the first time in week 10 ( ) through the downward cusum. up to and including the period which triggers the first signal. The ARL depends on the size of the change (including no change at all; the signal is then triggered by normal variation and results in a false alarm). In dairy production systems typically all observations are grouped in periods of one or more days. The average time to signal (ATS), calculated as ATS = ARL period length, is a more direct measure of the amount of time spent between the starting point and the first signal on a control chart. It is useful to distinguish between four notions of ATS: goal, target, design, and observed. The goal ATS is the desired average time between false alarms in practice. The target ATS is the desired average time between false alarms for observations that follow the assumed mathematical probability distribution. The placement of

7 A. de Vries, B.J. Conlin / Agricultural Systems 84 (2005) Fig. 4. Average run length (average number of observations to first signal) for a Shewhart control chart with 2.5- and 3-sigma control limits, assuming observations follow a normal distribution. SE = sigma = standard error. the control limits is determined by the target ATS and the assumed mathematical probability distribution. The design ATS is the average time between false alarms after the control limits have been placed and the observations exactly follow the assumed mathematical probability distribution. The design ATS is equal to the target ATS for continuous distributions with probability limits. The design ATS is different from the target ATS on Shewhart charts with symmetric control limits but with skewed mathematical probability distributions. Also, control limits can often not be designed for the exact target ATS on control charts for discrete distributions. The observed ATS is the actual ATS of the control chart in practice. The observed ATS can be different from the design ATS when the observations do not exactly follow the assumed mathematical probability distribution, through sampling error, and when the system has changed. The observed ATS after a real change is typically shorter when changes in the system are larger or when a shorter time between false alarms is accepted. Fig. 4 shows the observed ARL for various changes in the mean of a standard normal distribution (see Montgomery, 1997, p. 206). Note how the observed ARL is shorter for larger changes in the mean Description of the stochastic dairy herd simulation model The dynamic Monte Carlo dairy herd simulation model described and validated by devries (2001) was used to measure the performance of control charts for DFS and EDI to detect decreases in EDE. The model simulates individual dairy youngstock and cows from day to day through time. It contains functions for milk production, feed intake, disease, reproduction, and cow replacement. Stochastic elements are incorporated in the milk producing ability of animals, daily milk yield, occurrence and detection of estrus, occurrence and success of inseminations, errors in pregnancy diagnosis, occurrence and type of diseases, and death loss. The voluntary waiting period for a first insemination was 60 days after calving. Every observed estrus after the voluntary waiting period resulted in

8 324 A. de Vries, B.J. Conlin / Agricultural Systems 84 (2005) an insemination. The success of an insemination was resolved with pregnancy diagnosis 42 days later if the cow had not been observed in estrus earlier. All heifers were raised to enter the herd at calving. When a heifer entered the herd, the cow with the lowest future profitability (van Arendonk, 1984) was culled to maintain a constant herd size Design of simulation experiments A 1000-cow herd in steady state was created with 65% EDE. Two sets of four scenarios each were simulated (Fig. 5). Each scenario lasted a total of 4520 days and was repeated 400 times. In the first set, four scenarios were simulated to calculate reproductive statistics for DFS and EDI at steady state rates of 65%, 55%, 45%, and 35% EDE since Fig. 5. Overview of the rate of EDE in the simulation experiments. The horizontal bars ( ) show the rate of EDE in the simulation model over time. The first set of four scenarios was used to calculate steady state herd reproductive statistics at 65%, 55%, 45%, and 35% EDE. The four rates of EDE were implemented on day 1 and maintained for 4520 days. The first 2000 days were used to randomize the herd. Data were collected during days 2001 through The second set of four scenarios was used to obtain days to first signals on control charts. Days 2001 through 2840 were used to collect data to set up control charts at the start of day Estrous detection efficiency remained at 65% for the first 2840 days and either remained at 65% EDE or decreased to 55%, 45%, or 35% EDE during the remaining 1680 days. All eight scenarios were repeated 400 times.

9 A. de Vries, B.J. Conlin / Agricultural Systems 84 (2005) day 1. A startup period of 2000 days was used for each run to randomize the starting herd before data collection started. Data were then collected for the remaining 2520 days. Period length throughout the study was seven days. This set of scenarios was not used to evaluate the performance of control charts. In a second set of four scenarios, EDE was kept at 65% for the first 2840 days for all four scenarios. The first 2000 days were used to randomize the herds. The next 840 days were used to collect data to calculate parameters for DFS and EDI to set up the control charts at the end of day At the start of day 2841, either no decrease in EDE (same scenario as the first scenario in the first set), or a persistent step decrease to 55%, 45%, or 35% EDE was implemented. The decreases continued for 1680 days until the end of day Control limits were adjusted for each period depending on the subgroup size (for example, the number of cows that could be observed in estrus varied from period to period). Days to first signal were recorded for each control chart for each run. Most runs resulted in a signal within 1680 days. If no signal was triggered within 1680 days, then the days to first signal observation was censored at 1680 days Calculation of estrous detection index and choice of control charts Estrous detection index is the fraction of expected estruses that is observed in a period of seven days. It is calculated as: #observed estrusesðin periodþ Estrus detection indexðin periodþ ¼ #expected estrusesðin periodþ : ð1þ The number of observed estruses in a period is simply measured, but the number of expected estruses must be calculated. Because the length of the estrous cycle in dairy cows is assumed to be 21 days, every open cow is expected to have one estrus per 21 days. The total number of expected estruses in the herd in a period is then calculated as follows: Expected estrusesðin periodþ¼f# cow days after voluntary waiting period of cows confirmed open þ 0:6 # cow days of cows served with service result unknowng=21 ð2þ Every day a cow was in the herd was counted as a cow day. The constant 0.6 was the estimated fraction open cows in cows that were inseminated but establishment of pregnancy was still unknown. It was calculated as (1 conception rate) because the conception rate in the simulation model was on average 0.4. A fitting probability distribution of EDI was considered to be either a binomial or normal distribution. The binomial distribution is a natural candidate model when an expected estrus is considered a trial and an observed estrus a success. Estrous detection index is then the fraction successes. The Shewhart chart for the binomial distribution is called a P chart. The binomial cusum chart was also used. Because the

10 326 A. de Vries, B.J. Conlin / Agricultural Systems 84 (2005) Table 1 Overview of control charts that monitor location and dispersion applied to days to first service (DFS) and estrous detection index (EDI) by the assumed mathematical distribution Distribution Days to first service Estrous detection index Location Dispersion Location Dispersion Normal X (z-sigma) S (z-sigma) X (z-sigma) MR (z-sigma) X (z-sigma, suppl a ) S (probability) X (z-sigma, suppl) MR (probability) Cusum mean Cusum variance Cusum mean Cusum variance Binomial b P (z-sigma) P (probability) Cusum binomial Gamma Gamma (z-sigma) Gamma (probability) Cusum gamma Non-parametric Location (z-sigma) Dispersion (z-sigma) Cusum location Cusum dispersion a Three supplementary runs tests added. b Binomial charts monitor both location and dispersion. location (mean) and dispersion (variation) of the binomial distribution are dependent, only the fraction of successes needs to be monitored. The normal distribution was also considered because its location and dispersion are independent. This might allow for a better fit when the binomial assumptions do not hold. The mean of EDI was therefore also monitored with an X (individuals) chart and a normal mean cusum chart and the dispersion with an MR (moving range) chart and a normal variance cusum chart. Table 1 provides an overview of the various control charts used in this study Calculation of days to first service and choice of control charts Days to first service are the number of days between calving and a cowõs first insemination. When several cows have a first service in a period, a variety of statistics can be calculated, such as the mean, standard deviation, and the fraction cows that have fewer days to first service than the median. These statistics all describe one aspect of days to first service, but are in fact different measures of DFS. Table 6 in the Appendix A shows how these DFS statistics are calculated, depending on the type of control chart that is used. The mean of DFS per period was monitored by an X (subgroup means) chart and a normal mean cusum chart if a normal distribution was assumed. The dispersion per period was monitored separately by an S (standard deviation) chart and a normal variance chart. Montgomery (1997) prefers an S chart to the more traditional R (range) chart when sample size (the number of cows with a first service in the period) is variable, as was the case in this study. Previous analyses showed that the Gamma

11 A. de Vries, B.J. Conlin / Agricultural Systems 84 (2005) distribution was a good empirical model for the distribution of standard deviations of DFS. Therefore Shewhart charts based on the Gamma distribution (Gamma charts) and a Gamma cusum chart were added to monitor dispersion. The location and dispersion of DFS were also monitored with P charts and binomial cusum charts based on the non-parametric sign test as outlined by Amin et al. (1995). These control charts for the location required a count of the number of cows with fewer days to first service than the median number of days to first service in a period. Similarly, the P chart and binomial cusum chart for the dispersion were based on WestenbergÕs two-sample interquartile range test. This procedure required a count of the number of cows with fewer days to first service than the 25-percentile number of days to first service, or those with more days to first service than the 75-percentile in a period. Both counts were compared to the number of cows with a first service in the period using a binomial distribution with a 0.5 probability of success. Formulas for calculating the center line, the z-sigma ( z standard errors from the center line) control limits, and probability control limits are presented in Table 6 in the Appendix A. The design of the cusum charts is also described in the Appendix A Target ATS and the calculation of control limits Shewhart control limits were initially calculated with 3-sigma limits and later for a target ATS of 400 or 800 days. In the latter case, target false alarm rates were therefore equal to a = 1/(400 /7) = and a = 1/(800/7) = When the three supplementary runs test were added to the Shewhart charts (Fig. 2), the z-sigma constant was determined from Eq. (3) given a target ARL. The equation was estimated using simulation and holds for target ARL less or equal to 511 periods and control charts with both lower and upper control limits. Adjusted R 2 of Eq. (3) was Note that ARL = ATS/period length z 1 a=2 ¼ 2:385 þ 9: ARL 2: ARL 2 þ 3: ARL 3 : ð3þ For example, when the target ATS equaled 400 days, then for each individual control limit the design ARL = 400/7 2 = periods between false alarms. From Eq. (3) it then followed that both control limits were set at z 1 a/2 = 3.18 standard errors from the center line. The probability control limits for P charts were found by solving the equations in Table 6 for the target false alarm rate a. The exact target probability may not be obtainable, because the control limits for discrete distributions are necessarily integers. The control limits were chosen such that the probability of a false alarm was similar on both lower and upper control limits. Control limits for cusum charts were calculated with the ANYGETH program (Hawkins and Olwell, 1998). Initial results showed that the observed ATS at 65% EDE could significantly vary from the target ATS. The target ATS was later varied by trial and error such that the observed ATS at 65% EDE was similar to the goal ATS. Thus, the observed false

12 328 A. de Vries, B.J. Conlin / Agricultural Systems 84 (2005) alarm rates were similar. This procedure made a fair comparison of the performance of control charts to detect a true decrease in EDE possible Estimation of observed average time to first signal The observed ATS for each control chart and each scenario was calculated directly from the 400 days to first signal observations, provided each run resulted in a signal on the control chart within 1680 days. If the control chart failed to signal in at least one run, and therefore the observation was censored at 1680 days, a Weibull distribution was fit to the 400 ordered days to signal observations to estimate the observed ATS. The Weibull probability density function is defined in Eq. (4): WEIðxjc; bþ ¼c=b c c 1 expð ðx=bþ c Þ; ð4þ where x is the days to first signal, c is the shape parameter, and b is the scale parameter. The parameters c and b were estimated using the method described by Keats et al. (1997). Then ATS = E(x) =b C(c 1]/c) and VAR(x) =[b 2 (C([c + 2]/ c) C([c + 1]/)c 2 )]. The days to first signal distributions, including censored cases, between control charts were statistically compared using the non-parametric log-rank test calculated with Proc Lifetest in SAS (Allison, 1995). 3. Results 3.1. Steady state results The steady state results show the new level to which the various reproductive statistics will move if the decrease from 65% EDE is not signaled and corrected. Steady state averages and standard deviations for the major components of DFS and EDI at 35%, 45%, 55%, and 65% EDE were in agreement with expected trends (Table 2). The average DFS increased from 73.2 to 99.0 days when EDE decreased from 65% to 35%. The number of first services per week decreased from 18.3 to 15.8 while the number of first service days increased from 570 to 1074 per week. The EDI decreased from 0.50 to This decrease was caused by both a decrease in the number of observed estruses and an increase in the number of expected estruses. The variation in DFS and EDI changed as well when EDE decreased. The average standard deviation of DFS within weeks increased from 20.3 days (65% EDE) to 42.8 days (35% EDE). The effects of lower EDE rates also affected the variation between DFS averages, which increased from 5.1 to 11.5 days. The variation in the number of first service days also increased, but the variation in the number of first services remained fairly constant. The variation in EDI decreased from a standard deviation of (65% EDE) to (35% EDE). The variation in the number of observed estruses decreased as well, but the variation in the number of expected estruses increased. These changes in variation suggest that it is important to monitor the dispersion as well as the location of reproductive measures.

13 A. de Vries, B.J. Conlin / Agricultural Systems 84 (2005) Table 2 Steady state reproductive statistics for four levels of estrous detection efficiency. Herd size is 1000 cows. Period length is seven days Reproductive statistic Estrous detection efficiency 65% 55% 45% 35% Average days to first service 73.2 ± ± ± ± 11.5 SD days to first service a 20.3 ± ± ± ± 11.1 # First services 18.3 ± ± ± ± 3.9 # First service days 570 ± ± ± ± 77 Estrous detection index 0.50 ± ± ± ± 0.04 # Observed estruses 43.3 ± ± ± ± 5.5 # Expected estruses 87.4 ± ± ± ± 4.9 a Average standard deviation within periods of seven days Herd dynamics after a decrease in estrous detection efficiency Most reproductive statistics would not immediately reach their new steady state level after a decrease in EDE, as a result of lag and momentum. Figs. 6 8 suggest that it could take several years before the new steady state level was reached. An exception was EDI (Fig. 8), which immediately reflected the new, lower, EDE rate at the start of week 120 (day 2841). The number of first service days (Fig. 7) and the number of expected estruses (Fig. 8) fluctuated systematically together for several years, until they reached their new steady state at the lower EDE rate. The decrease in EDE also affected both the fractions open and lactating cows (not shown). The fraction open cows increased immediately for about nine months, after which it dropped and started fluctuating in correspondence with the number of expected estruses until it reached a higher steady state level. The fraction lactating cows started increasing only seven months after the start of the decrease in EDE, when fewer cows were dried off because fewer cows had become pregnant. The fraction Fig. 6. Course of the average and standard deviation (SD) of days to first service during 360 weeks with a decrease from 65% EDE to 45% EDE at the start of week 120 (day 2841). Results from one run.

14 330 A. de Vries, B.J. Conlin / Agricultural Systems 84 (2005) Fig. 7. Course of the number of first service days () and the number of first services ( ) per week during 360 weeks with a decrease from 65% EDE to 45% EDE at the start of week 120 (day 2841). Results from one run. Fig. 8. Course of the number of expected estruses () per week and estrous detection index ( ) during 360 weeks with a decrease from 65% EDE to 45% EDE at the start of week 120 (day 2841). Results from one run. of lactating cows reached its peak also at about nine months after the start of the decrease in EDE, after which it dropped and started fluctuating until it reached a higher steady state level. The duration of the delay, systematic fluctuations, and normal variation in herd performance after a decrease in EDE all affect the performance of the control charts Performance of Shewhart control charts with 3-sigma control limits and supplementary runs tests For normally distributed observations, the design ATS on the X (for DFS) and X (for EDI) control charts with 3-sigma control limits is equal to 2593 days (z /2 = 3, design ARL = weeks = 2593 days, Table 3). When the three supplementary runs tests are applied to these X and X charts, and assuming normally distributed observations, the design ATS decreases to 555 days (79.3 weeks,

15 A. de Vries, B.J. Conlin / Agricultural Systems 84 (2005) Table 3 Average time to signal (ATS, in days) for Shewhart control charts with control limits set at three standard errors (3-sigma) from the center line Shewhart control chart Observed ATS Design ATS a 65% EDE b 55% EDE 45% EDE 35% EDE Days to first service X (3-sigma) X (3-sigma, suppl. c ) S (3-sigma) Estrous detection index P (3-sigma) X (3-sigma) X (3-sigma, suppl.) MR (3-sigma) d ,982 a Design ATS is the average number of days between false alarms assuming a binomial distribution for the P-chart and a normal distribution for all other control charts. The design ATS for supplementary runs tests, S and MR charts were found using simulation. b The observed ATS at 65% EDE is the observed average number of days between false alarms. The observed ATS at 55%, 45%, and 35% EDE is the average number of days until the decreased EDE rate is first signaled. c Three supplementary runs tests added. d MR control limits set at 0 and ( )/ moving average. only 21% of the target ATS), because applying more tests to the same observations increases the false alarm rate. Using steady state results for the dairy herd (Table 2), if EDI followed a binomial distribution with 87 expected estruses in a week and a 0.5 probability of successful detection of estrus (the average EDI), then the design ATS on the P chart with 3-sigma control limits would be 2823 days (403.3 weeks). For the 3-sigma S-chart, if DFS followed a normal distribution with mean 73.2 and variance (again taken from Table 2), the design ATS would equal 1736 days (248 weeks). The MR chart for EDI, assuming a normal distribution with mean 0.50 and variance , has a design ATS equal to 763 days (109 weeks). Thus, the design ATS varied significantly for the traditional Shewhart charts with 3-sigma limits and was often much shorter than the expected 2593 days. The observed average times between false alarms (ATS 65% ) could be quite different from these design ATS. The observed ATS 65% was shorter than the design ATS for all control charts, except for the X chart for EDI. The observed ATS 65% of the X chart for DFS and the X and MR charts for EDI were within 69% (1800/2593) to 114% (2949/2593) of the design ATS. The observed ATS 65% for the S chart and the control charts with supplementary runs tests was much shorter than expected, about 100 days between false alarms (<4% of the target ATS). Large differences between the observed and design ATS were primarily observed because DFS and EDI did not follow exact normal or binomial distributions. As expected, larger decreases in EDE were signaled earlier (having shorter ATS) with all control charts, except with the MR chart. The variation in EDI was less at lower EDE rates, but the traditional MR chart does not have a lower control limit to

16 332 A. de Vries, B.J. Conlin / Agricultural Systems 84 (2005) signal these lower levels of variation. Therefore, a signal could only be triggered by the upper control limit, falsely suggesting an increase in variation. The observed ATS after a decrease to 35% EDE (ATS 35% ) was at least 45 days for Shewhart charts applied to DFS and eight days for Shewhart charts applied to EDI. The standard error of each observed ATS in the study was approximately equal to ATS/ , because the distribution of days to signal is approximately geometric. Thus an approximate 95% confidence interval for each observed ATS was [0.9 ATS, 1.1 ATS] Performance of control charts with target ATS equal to 400 or 800 days Tables 4 and 5 show that the observed ATS 65% was often significantly shorter than the target ATS of 400 or 800 days (columns 2 and 3). Especially the X and X charts with supplementary runs tests, the control charts based on the normal variance for DFS, and the P charts for EDI led to ATS 65% that were much shorter than the target ATS. The observed ATS 65% on the S charts was about 40 to 60 days (<11% of the target ATS), depending on the target ATS. The observed ATS 65% of the gamma charts was much closer to their target ATS, which showed that the distribution of the standard deviation of DFS was better modeled by a gamma distribution. The observed ATS 65% for the P charts applied to EDI was approximately only 20% of the target ATS, which indicated that the binomial distribution was not a good model for EDI. The observed ATS 65% of the non-parametric charts was similar to the target ATS, as expected. This result confirmed their more robust false alarm rates compared to the parametric charts when observations do not exactly follow the assumed mathematical probability distribution Performance of control charts with goal ATS equal to 400 or 800 days The target ATS was varied for each control chart (and therefore the control limits were varied) until the observed ATS 65% was similar to the goal ATS of 400 or 800 days. The target ATS that accomplished this task best is called best-target ATS (column 4 in Tables 4 and 5). If the average observed ATS 65% was shorter than the target ATS of 400 or 800 days (previous section), then the best-target ATS had to be longer than 400 or 800 days (resulting in wider control limits) in order to increase the average observed ATS 65% to obtain the goal of days. The best-target ATS for the control charts monitoring the normal variance and the control charts based on the binomial distribution for EDI were consequently very long. For example, a best-target ATS for the S (z-sigma) chart was 6,780,231 days, which let to control limits set at 4.92 standard errors from the center line to obtain an observed ATS 65% of 800 days. For normally distributed observations, the control limits would be set at only 2.62 standard errors from the center line to obtain an observed ATS 65% of 800 days. The best-target ATS typically resulted in an observed ATS 65% within one or two days from the goal ATS (column 5). The observed ATS 65% for non-parametric dispersion charts were up to 50 days longer or shorter than the goal ATS. These larger

17 A. de Vries, B.J. Conlin / Agricultural Systems 84 (2005) Table 4 Average time to first signal (ATS, in days) on control charts for days to first service. Estrous detection efficiency (EDE) remained at 65% or decreased to 55%, 45%, or 35%. a Control chart b Goal ATS Observed ATS when target ATS = goal ATS Best-target ATS Observed ATS with besttarget ATS EDE 65% 65% 65% 65% 55% 45% 35% Shewhart control charts X (z-sigma) X (z-sigma, suppl.) S (z-sigma) , S (probability) , Gamma (z-sigma) Gamma (probability) NP location (z-sigma) c NP dispersion (z-sigma) Cusum control charts Normal mean Gamma Normal variance , NP location NP dispersion Shewhart control charts X (z-sigma) X (z-sigma, suppl.) S (z-sigma) , S (probability) , Gamma (z-sigma) Gamma (probability) NP location (z-sigma) NP dispersion (z-sigma) Cusum control charts Normal mean Gamma Normal variance , NP location NP dispersion a Goal ATS is the desired average time between false alarms in practice, target ATS is the average time between false alarms when the observations follow the assumed mathematical probability distribution, best-target ATS is the target ATS such that the goal ATS and the observed ATS at 65% EDE are similar, observed ATS is the observed average time to a first signal in practice. b z-sigma: control limits set at z standard errors from the center line. Probability: control limits set with a similar probability of a false alarm on either control limit. Suppl.: three supplementary runs tests added. c NP = non-parametric control chart. deviations were the results of the limited number of design ATS that are possible with discrete observations. An observed ATS 65% equal to the goal ATS could therefore not always be exactly obtained.

18 334 A. de Vries, B.J. Conlin / Agricultural Systems 84 (2005) Table 5 Average time to first signal (ATS, in days) on control charts for estrous detection index. Estrous detection efficiency (EDE) remained at 65% or decreased to 55%, 45%, or 35% a Control chart b Goal ATS Observed ATS when target ATS = goal ATS Best-target ATS Observed ATS with besttarget ATS EDE 65% 65% 65% 65% 55% 45% 35% Shewhart control charts X (z-sigma) X (z-sigma, suppl.) P (z-sigma) P (probability) MR (z-sigma) MR (probability) Cusum control charts Normal mean Binomial Normal variance Shewhart control charts X (z-sigma) X (z-sigma, suppl.) P (z-sigma) , P (probability) , MR (z-sigma) ,511 MR (probability) Cusum control charts Normal mean Binomial , Normal variance a Goal ATS is the desired average time between false alarms in practice, target ATS is the average time between false alarms when the observations follow the assumed mathematical probability distribution, best-target ATS is the target ATS such that the goal ATS and the observed ATS at 65% EDE are similar, observed ATS is the observed average time to a first signal in practice. b z-sigma: control limits set at z standard errors from the center line. Probability: control limits set with a similar probability of a false alarm on either control limit. Suppl.: three supplementary runs tests added. The goal ATS of 800 days could not be obtained with the X chart with supplementary runs tests. Test 4 (nine observations below or above the center line) is independent of the placement of the control limits and therefore is not affected by the size of the target ATS. Simulation showed that the maximum observed ATS 65% with the supplementary runs tests was 637 days for DFS and 585 days for EDI. In these cases the control limits were extremely wide and all signals were triggered by Test 4. The goal ATS of 800 days could also not be obtained with the MR (probability) chart because the moving range of EDI was often 0, which triggered too many signals on the positive lower control limit.

19 A. de Vries, B.J. Conlin / Agricultural Systems 84 (2005) When all false alarm rates had been set similar to the goal, larger decreases in EDE were signaled earlier with all control charts, except with the MR charts. The observed ATS at 55%, 45% and 35% were longer when the goal ATS was longer in all cases. Thus the price paid for a lower false alarm rate (a longer goal ATS) was a longer time to first signal when EDE truly had decreased. Proc Lifetest analyses showed that differences in observed ATS of a few days among control charts were typically statistically significant. When DFS was monitored on control charts, decreases to 55% and 45% EDE were signaled the earliest with the normal cusum charts (Table 4, columns 6 and 7). For an average time between false alarms of 400 days, the minimum observed ATS 55% and ATS 45% were 85 and 63 days, respectively. Other charts signaled these decreases in EDE significantly later, with the exception of the X (z-sigma) chart with supplementary runs tests and a goal ATS of 400 days, which had an observed ATS 55% of 87 days (P ). For an average time between false alarms of 800 days, the minimum observed ATS 55% and ATS 45% were a little longer, respectively 97 and 69 days. The minimum observed ATS 35% for DFS were 55 days with the X (zsigma) chart (but not significantly different from the normal mean cusum chart, P ), and 59 days with the normal mean cusum chart for the goal ATS of 400 and 800 days, respectively. The non-parametric charts for DFS generally signaled decreases in EDE significantly later than the earliest parametric control charts. When EDI was charted, decreases to 55% EDE were the earliest signaled in 38 (goal ATS was 400 days) and 50 days (goal ATS was 800 days) with the normal mean cusum chart. The binomial cusum chart signaled decreases to 45% EDE the earliest in 14 and 16 days, respectively. This was just one or two days earlier than the normal cusum chart, although the difference was significant (P ). The minimum observed ATS 35% for EDI for either goal ATS was eight days, which was obtained with many control charts. The design of the experiments with period length of seven days made signaling of a decrease in EDE earlier than seven days not possible. Finally, for each scenario, the earliest control chart for EDI signaled the decrease significantly earlier than the earliest control chart for DFS. 4. Discussion In this study we used a complex Monte Carlo simulation model of a dairy herd to study the performance of statistical quality control charts under practical conditions. Control charts are useful in situations where it is too costly to explore in detail all the data generated by the system that is monitored. This is increasingly the case on modern dairy farms were more data is becoming available from sensors that observe the status and behavior of cows, such as activity data, and production data, such as from the Dairy Herd Improvement Association, daily milk weights, and milk conductivity data (Pietersma et al., 1998). Moreover, herds are growing larger in the US, increasing the volume of available data. Only when there is sufficient evidence that an aspect of the dairy production system has fundamentally changed is an investigation (with its associated costs and time requirement) of the potential causes warranted. The