August 24, Jean-Philippe Mathevet IAQG Performance Stream Leader SAFRAN Paris, France. Re: 2014 Supplier Performance Data Summary

Transcription

1 Page 1 of 27 August 24, 2105 Jean-Philippe Mathevet IAQG Performance Stream Leader SAFRAN Paris, France Re: 2014 Supplier Performance Data Summary Dear Jean-Philippe: The following report is a summary of the 2014 and 2010/2012/2013 supplier performance data that was provided. The data represents only a small random sample of the aerospace suppliers, and the data was collected and organized by a 3rd party, CAPS Research. The report may be distributed to any or all members of the IAQG. Specifically, the analysis includes the data in the Excel files called "IAQG Survey 2015 MASTER WORKBOOK for IAQG_USE_Phase xlsx," and "IAQG Survey 2013 MASTER from CAPS for IAQG use 2010.xlsx." The report begins with some background and definitions. The remaining sections illustrate the results and conclusions in more detail. The report is organized in the following sections: 1. Background... Page 2 2. Definitions... Page 2 3. Confidence Interval and Statistical Test Concepts... Page 3 4. Analysis of 2014 IER... Page 5 5. Analysis of 2014 OTD... Page /2012/2013 vs IER... Page /2012/2013 vs OTD... Page TS1 Analysis... Page TS2 Analysis... Page TS3 Analysis... Page TS4 Analysis... Page Yearly Comparisons by Technology Segment... Page Sampling Strategy... Page Sample Size... Page Performance Indicators... Page 24 APPENDIX... Page 27

2 Page 2 of BACKGROUND The IAQG periodically collects suppliers performance data (i.e. Item Escape Rates and Late Deliveries) in order to publish a Performance Index. A few questions regarding performance include: 1. Do the 26 aerospace primes see improvement in performance of the 20,000 estimated supplier sites (14,000 certified (9100) + 6,000 others)? 2. Are Technology Segment, Certification Status, Statistical Process Control (SPC) use, and/or Nadcap accreditation predictors of performance? 3. Is there evidence that performance is improving? 4. Is the current sample size adequate for identifying differences among Technology Segments, Certification Status, Statistical Process Control (SPC) use, and Nadcap accreditation? 5. Are there other important metrics that better differentiate suppliers with regard to quality and service? 2. DEFINITIONS The two performance features currently being collected for this IAQG supplier performance study are the Item Escape Rate (IER) and On Time Delivery (OTD). It is important that all the participants in the study use precisely the same definitions, so for clarity, the definitions are provided here. The source is the Delivery Metrics Definition Guidance section of the IAQG Supply Chain Management Handbook (SCMH). First, an "escape" is a non-conforming item that has reached a customer. From this notion, one of two metrics is typically captured. One metric is the Item Escape Rate (conformity for items) and the other is the Defects per Unit (DPU) (conformity for systems). The SCMH states that if more than 1000 items are delivered per year, IER should be used. Otherwise, DPU should be reported. In the 2012 survey data, the vast majority of polled suppliers indicated an IER, so in the 2014 survey, respondents were asked for an IER (rather than a DPU) metric. Last year, there were only 262 surveyed suppliers. This year, due to far better participation, there were 1084 surveyed suppliers. Of the 1084 surveyed suppliers, 8 did not have an associated IER, but the remaining 1076 did. Note that throughout this report, "n" refers to the sample size of a group.

3 Page 3 of 27 The following is the definition for the IER: Item Escape Rate (IER) IER = 1,000,000 Number of non conforming items Number of items delivered by company so that IER is expressed in parts per million (ppm). Note that "Number of nonconforming items" refers to items that are under the suppliers' liability. Defects caused by transporters or customers should not be counted. The following is the definition for the OTD: On Time Delivery (OTD) OTD = Number of Purchase Order items due & delivered on time in the period Number of Purchase Order items due in the period 3. CONFIDENCE INTERVAL & STATISTICAL TEST CONCEPTS Confidence Intervals Confidence intervals are used to express uncertainty in the estimation of (unknown) population statistics. Using sample data, we can calculate a point estimate (single value) but the confidence interval is useful to quantify the amount of uncertainty in the estimate. The confidence interval is the range which is expected to trap the true (unknown) value with a specified probability. For example, suppose the estimated median from our sample data is 50 and the 95% confidence interval is calculated to be (40,60). Then we can say that we are 95% confident that the interval bounded by 40 and 60 traps the true (unknown) population median. The precision (width) of the confidence interval is a function of the amount of data in our sample, the variability in the data, and the confidence level. A 99% confidence interval will be wider (less precise) than a 95% confidence interval (all other factors being equal) since it must trap the true median with higher probability. Confidence intervals specify a coverage (confidence/uncertainty) probability. A coverage probability of 95% is most commonly specified although other coverage probabilities such as 90% or 99% are sometimes used. The coverage probability will be higher (such as 99%) when we have a low tolerance for the risk that the interval does not include the true (unknown) value. So utilizing a 99% confidence interval will have only a 1% chance of the interval not including the true value. This report uses 95% confidence intervals, since it is the most common coverage probability used in typical statistical analyses.

4 Page 4 of 27 Statistical/Hypothesis Tests Throughout the analyses, several statistical tests were performed to compare groups. Usually, when comparing 2 groups, the Mann-Whitney test was used (more detail to follow), and when comparing 3 or more groups, the Kruskal-Wallis test was used. The Mann-Whitney test (also called the 2-sample rank test) tests the equality of two population medians (i.e. middle values). This "non-parametric" test is an alternative to the 2- sample t parametric test for testing the equality of two population means (which assumes normally distributed data in each group). Since for non-normal data, the means are not a robust measure of central tendency, we test the medians, which are not overly influenced by extreme values). The null hypothesis (initial belief) is: H 0 : Median 1 = Median 2 (in words, Median of group 1 = Median of group 2) The alternate hypothesis is typically: H 1 : Median 1 not = Median 2 The Mann-Whitney procedure involves combining the samples and ranking the data in order. Where ties are present (2 identical values), the average rank is assigned to each. A test statistic, W, is computed which is the sum of the ranks in the first sample. If there is enough evidence to reject the initial (null) hypothesis, then we conclude that the two groups have statistically different medians. Statistical/Hypothesis Tests vs. Confidence Intervals In hypothesis testing, testing for statistically significant differences is usually done using a test statistic (hypothesis testing) or p-value. However, conceptually, the relative relationship between a point estimate from one group and the confidence interval estimate for the other group is indicative of whether a statistical significant difference exists or not. Typically, if a significant difference is concluded, then at least one sample's point estimate falls outside the estimated confidence interval for the other sample estimate. If no significant difference is concluded, then typically at least one sample's point estimate falls inside of the estimated confidence interval for the other sample estimate. The methodology used to compute confidence intervals is different than that used to conclude whether a significant difference actually exists at the specified confidence level. Therefore, while the relationship between point estimates from one group with the confidence interval of the other group is usually indicative (and consistent with the results) of the hypothesis test, some exceptions will be encountered. In general, and one cannot utilize confidence intervals to determine whether a statistical difference exists.

5 Frequency Page 5 of ANALYSIS OF 2014 Item Escape Rate (IER): The 2014 IER Data In order to use most of the common statistical methods, the analyst should be able to identify a distribution (curve) that describes the data. In cases that we cannot, we often try transforming the data, such as taking a natural log, a power transformation, a trigonometric transformation, or some combination of those. Often, a transformation will help the analyst find a distribution that describes the data--and some transformations can make the data appear "bell-shaped" (Normal distribution). In the case of the 2014 IER data, no data transformation or well-known distribution describes the data due to a large fraction of zeros in the data. In fact, of the 1076 IER responses, 249 (23%) had IER values of zero. The graph below indicates the nature of the IER data. Note that the horizontal scale was truncated. In fact, there were 14 suppliers with IER of 30% or higher. One supplier had an IER of over 90% (913,043 ppm). 600 Histogram of Item Escape Rate (IER) ppm IERppm As a result of the dataset, typical (parametric) statistical methods are not applicable, since the data violates the assumptions required for those methods. Instead, non-parametric statistical methods were used. Non-parametric methods do not require many assumptions of the data. They are ideal for data sets such as the IER data. The down-side of non-parametric methods is that they are usually not as good as parametric methods (such as t-tests, ANOVA, etc.) at identifying differences between groups if they exist. In statistical language, they are not as "powerful" at detecting differences.

6 Page 6 of 27 In the analyses that follow, we will not discuss means (averages). The reason is that one bad supplier in the sample can have an overwhelming influence on the average. For example, in the 2014 IER data, one of the suppliers has an IER of 913,043 ppm (91.3%). As a result, the average 2014 IER is 20,459 ppm. The median (50th percentile) however, is only 1,980 ppm. From the suppliers who provided IER data, half of them have IER values less than 1,980 ppm. As a result of the highly skewed and unstable average estimates, we will discuss the median (50th percentile) values instead. Keep in mind that the variability in supplier IERs is huge. While 23% had an IER of 0 ppm, 57 sampled suppliers had values exceeding 100,000 ppm (10%). That is, 5.3% of sampled suppliers (supplying IER) had IER values exceeding 10%. In the graphic below (salary data from a large city in 1991), the mean is $120,000 (not indicative of the typical earner), and $52,000 is the median. Half the workers earn less than $52,000, whereas roughly 80% learn less than average. In cases where the data is not symmetric, the median can be more useful. Graphics In this section and the following sections, some graphics of the IER and OTD data will be provided. Some the of the graphics simply show the raw data, where each data point is indicated by a dot on the chart. However, sometimes, the data has been summarized in a "Boxplot."

7 Measurements Page 7 of 27 A boxplot helps to visualize the location and variation of measured characteristics (such as chrome weight). An example boxplot is shown below and the key features are described. Boxplot of Measurements 200 Points "far" from the bulk of the data 150 Maximum (excluding "far" points) 100 The box containts the "inner" 50% of the data 85 75th Percentile Median (Middle Value) 25th Percentile Minimum Value The measured values are shown on the y-axis (vertical axis). The box (shaded) traps the inner half (50%) of all the data. So, half the data falls between 46 and 85. The box consists of the middle half of the data. The smallest value in the data set (Minimum Value) is 20 and is at the bottom of the line (called a whisker) that extends below the box. The horizontal line at the bottom of the box is called the 25 th percentile. In this data set the value is 46 and it represents the value at which 25% of the data falls below. The horizontal line inside the box is called the median (also called the 50 th percentile). In this data set the median is 60 and it represents the value at which 50% of the data falls below (the other 50% of the data falls above). So, half the data falls below 60. The horizontal line at the top of the box is called the 75 th percentile. In this data set the value is 85 and it represents the value at which 75% of the data falls below (and 25% above). In the boxplot above a whisker extends above the box and stops just below a value of 150. The end of the whisker is the largest value excluding some points which are larger but are considered far from the rest of the data. The group of asterisks at the top represent data points that are considered far from the rest of the data.

8 IER ppm Page 8 of IER by Technology Segment Since the data is not bell-shaped (nor does it follow any other well-known distribution) a nonparametric method called the Kruskall-Wallis test was used to compare IER Medians by Technology Segment. The technology segments are: TS1: manufactured parts; TS2: subsystems and equipments; TS3: raw materials; TS4: hardware or technical products. In 79 instances, a supplier did not indicate a Technology Segment. Statistically, TS2 and TS3 had the highest medians (worst), and TS1 and TS4 had the lowest (best). TS1 and TS4 were statistically indistinguishable from each other. TS2 and TS3 were indistinguishable from each other. However, TS2 and TS3 were both statistically worse than TS4. TS2 was statistically worse than TS1. The test used to make all pair-wise comparisons was the Mann-Whitney test. The graphic below illustrates the data, and the sample sizes are shown above the horizontal axis. The scale was adjusted to better show the comparisons IER by Tech Segment with 95% Confidence Intervals for Medians n=79 n=487 n=260 n=147 n=103 blank TS 1 TS 2 TS 3 TS 4 Technology Segment The medians of each group are shown along with colored 95% confidence intervals about the medians. TS1 & TS4 are not statistically different TS2 & TS3 are not statistically different TS2 is statistically worse than TS1 and TS4 TS3 is statistically worse than TS4 "n" indicates the sample size in each Technology Segment

9 IERppm Page 9 of 27 Note that the variability in each Technology Segment is large. Each segment included suppliers with IER values of 0 and IER values that exceeded 300,000 ppm. Reasons for the sometimes high IER should be investigated. Non-conforming product deliveries violate contracts and cause customers productivity and profit losses--and potentially high risks when unacceptable products are not detected in the supply base. We recommend that root causes be investigated. Other industries, such as foods/beverages, medical devices, and pharmaceuticals could not afford to tolerate these escape rates IER by Certification When the 2014 IER data was compared by whether suppliers held the various certifications, the results were disappointing. The uncertified suppliers had a lower median IER (1178 ppm) than the certified suppliers (2034 ppm), but the difference was not statistically significant. Comparisons of medians were used for this conclusion. If one erroneously compared the mean IER values between certified and uncertified of means, one would still find no evidence of any difference in the IER values between certified and uncertified suppliers. In the 2014 sample, 150 suppliers were indicated as not certified, 859 were indicated as certified, and 67 suppliers were left as "unknown" or blank. The graph below includes all data, but the scale was shortened so that the bulk of the data could be viewed. IER Boxplot by 9100 Certification with 95% Confidence Intervals for Medians No Unknown Yes n=150 n=66 n=859 Results exclude rows where 9100 Cert="blank" Certification There is no evidence that 9100 certification is associated with lower IERs.

10 IER ppm Page 10 of IER by Statistical Process Control (SPC) Use In the 2014 aerospace supplier survey, companies were asked whether they use SPC. The use of SPC was a highly significant predictor of IER--and the biggest predictor of IER in this database. Statistically, suppliers responding yes to the use of SPC had significantly lower IERs than suppliers indicating no to the use of SPC. The difference in medians was significant at the level. 77 indicated yes for SPC use, 240 indicated no, and the rest were missing or unknown. The graphic below indicates the result. Note that the scale was shortened so that the majority of the data could be observed. Boxplot of IER ppm by SPC Use with 95% Confidence Intervals for Medians SPC_No SPC_Yes n=240 n=77 The use of SPC is significantly associated with lower IERs

11 Page 11 of IER by NADCAP Accreditation In the 2014 aerospace supplier survey, companies were asked whether suppliers' special processes were covered by NADCAP accreditation. 293 responded "yes," 368 responded "no," and the remaining were recorded as "unknown" or blank. Statistically, there was no difference between certified versus uncertified group. In fact, both groups had nearly identical medians. The certified group had a median IER of 1935 ppm, and the uncertified group had a median IER of 1913 ppm. 5. ANALYSIS OF 2014 On Time Delivery (OTD): The 2014 OTD Data In order to use many common statistical methods, the analyst should be able to identify a distribution (curve) that describes the data. In this case (OTD), the data is proportion data (data bounded by 0 and 1 or 0 and 100%). So, the data is not bell-shaped (Normal, Gaussian), but often a transformation can adjust the data so that it does look bell-shaped. The two most common transformations for this type of data are the "Logit" and the "Arcsine Square Root." Both transformations were tried, however, due to the large fraction of 100% OTD, neither transformation was effective. Also, due to the large proportion of 100% OTD, no common statistical distribution described the data. Of the 1084 supplier results, 1079 had OTD responses and 5 were missing. Also, of the 1079 responses, 192 (17.8%) had OTD of 100%. So, as with the IER data, non-parametric methods were used OTD by Technology Segment Technology Segment was a predictor of OTD%. Technology Segment 4 had a statistically higher OTD% median than Technology Segments 2 and 3. Technology Segment 1 had a statistically higher OTD% than Technology Segment 3. All other comparisons were statistically the same. The following boxplot illustrates the results:

12 OTD% Page 12 of 27 OTD% by Technology Segment with 95% Confidence Intervals for Medians % 91.2% 96.7% 85.4% TS 1 TS 2 TS 3 TS 4 n=487 n=263 n=147 n=103 Technology Segment 2014 OTD by Certification 9100 Certification Status was potentially a predictor of OTD%, but in a disappointing way. At a 0.07 level of significance, the Uncertified group had a better OTD%, but at the 0.05 level of significance, there was not a difference. All the comparisons in this report are made at the 0.05 level of significance. The Uncertified group had a median OTD% of 95.8%, and the Certified group had a median OTD% of 91.7%. The boxplot below illustrates the result.

13 On Time Delivery (OTD%) Page 13 of 27 OTD by 9100 Certification with 95% Confidence Intervals for Medians % % _No 9100_Yes n=150 n= OTD by SPC The survey questioned whether SPC used. Statistically, whether SPC was used (or not) did not impact OTD. The median for the SPC group was 97% (n = 77) and the median for those responding "no SPC" was 99% (n = 240) OTD by NADCAP Accreditation NADCAP Accreditation was not a predictor of OTD%. There was no evidence of a statistically significant difference in OTD% by NADCAP Accreditation Status. The "yes NADCAP" group had a median OTD of 92% (n = 291) and the "no NADCAP" group had a median OTD of 94% (n = 366) vs 2012 vs 2013 vs 2014 IER DATA Data was collected and consolidated by CAPS Research in At that time, not as much information was captured, but there is data for OTD and IER by Technology Segment. The 2010 dataset had 187 respondents as opposed to the 233 respondents for 2012, 262 for 2013, and 1084 for With regard to the IER metric, 2014 is statistically better than The median IER for 2010 is 4,296 ppm and the median for 2013 is 1,980 ppm. However, there was no difference

14 IER ppm Page 14 of 27 between the IERs of 2012, 2013, and The test used was the Mann-Whitney Test. The graph below illustrates the data. Instead of the individual values, a boxplot of each group was used. In the following graphic, there is no vertical line protruding from the bottom of the box, because there was a large fraction of zeros in the data, so the 25th percentile is close to zero. Also, the vertical scale on the graph was modified, since every year had sampled suppliers with IER values exceeding ppm (30%), and the graph was too difficult to read. In fact, 2010, 2013, and 2014 surveys included suppliers with IER exceeding 500,000 ppm (more than 50% non-conforming units escaping) IER with 95% Confidence Intervals for Medians IER 2012 IER 2013 IER 2014 IER n=187 n=212 n=256 n= , 2013, 2014 are significantly lower than 2010 IERs 2012, 2013, 2014 are not statistically different from each other

15 On Time Delivery (OTD%) Page 15 of vs 2012 vs 2013 vs 2014 OTD DATA Data was collected and consolidated by CAPS Research in At that time, not as much information was captured, but there is data for OTD and IER by Technology Segment. The 2010 dataset had 187 respondents as opposed to the 233 respondents for 2012, 262 for 2013, and 1084 for With regard to the OTD metric, 2014 is statistically better than 2010, 2012, and OTD by Year with 95% Confidence Intervals for Medians % 87.2% 91.6% % n=187 n=233 n=249 n= , 2013, and 2014 are significantly higher than 2010 OTDs 2014 is statistically better than all previous years 8. TECHNOLOGY SEGMENT 1 ANALYSIS Of the 1084 responses, 490 are given as TS1. With regard to 9100 Certification, the median IER ppm for the certified group was 1570 ppm. The median IER ppm for the uncertified group was 1178, but the difference was not statistically significant. Certification had no association with OTD%. The TS1 OTD for 9100 certified group was 94% (n = 408) and it was 97% (n = 58) for the non-certified group. With regard to the use of SPC, the median IER ppm for the users of SPC was 242. The median IER ppm for the SPC = "no" group was 1610, and the difference is highly statistically

16 IER ppm Page 16 of 27 significant (at the level). Only the TS1 group had a substantial number of "yes" responses for SPC usage. SPC was associated with slightly higher OTD% median (99.2% vs. 96.1%), but the difference is only significant at the 0.1 level, and not the 0.05 level. TS1: IER by SPC Use with 95% Confidence Interval for SPC=YES SPC_No SPC_Yes n=160 n=60 With regard to NADCAP, the "yes" group had a higher median IER ppm (1572) than the "no" group (1170 ppm), but the difference was not statistically significant. NADCAP was associated with slightly higher OTD% median (95.8% vs. 94.4%), but the difference is only significant at the 0.1 level, and not the 0.05 level. 9. TECHNOLOGY SEGMENT 2 ANALYSIS Of the 1084 responses, 265 are given as TS2. With regard to 9100 Certification, the median IER ppm for the certified group was The median IER ppm for the uncertified group was 1404, but the difference was not statistically significant. With regard to OTD%, the uncertified group had a slightly better median OTD% than the certified group (93.3% vs. 90.5%), but the difference was only significant at a 0.1 level of significance--and not the 0.05 level. With regard to the use of SPC, only 3 respondents indicated "yes," so the sample size is too small. The median IER was much smaller for the SPC "yes" group than the no, but a sample size of 3 is too small for a legitimate comparison.

17 On Time Delivery (OTD%) Page 17 of 27 With regard to IER and NADCAP, there was essentially no difference between the "yes" and "no" groups. The "yes" group had a median of 4129 (n = 43), and the "no" group had a median of 6011 (n = 84). For OTD%, the non-nadcap group had a significantly higher OTD% (97.7%) than the certified group (87.0%). Tech Seg 2: OTD% by NADCAP with 95% Confidence Intervals NADCAP_No NADCAP_Yes n=83 n= TECHNOLOGY SEGMENT 3 ANALYSIS Of the 1084 responses, 147 are given as TS3. With regard to 9100 Certification, only 10 indicated "no" for certification, so the statistical test (Mann-Whitney) was not powerful enough to detect a difference between the certified and non-certified group. With regard to the use of SPC, only 13 respondents indicated "yes," and only 19 indicated "no." Again, due to small sample sizes (and both groups having a very low median IER), statistical comparisons could not detect any difference between the two groups. With regard to NADCAP, there was no statistical difference between the "yes" and "no" groups for IER, but the NADCAP group was significantly worse for OTD. The summary is as follows:

18 Page 18 of 27 TS3 IER Median Yes NADCAP = 925 ppm, n = 51 TS3 IER Median No NADCAP = 2791 ppm, n = 47 TS3 OTD Median Yes NADCAP = 78%, n = 51 TS3 OTD Median No NADCAP = 94%, n = TECHNOLOGY SEGMENT 4 ANALYSIS Of the 1084 responses, 103 are given as TS4. With regard to 9100 Certification, the median IER ppm for the certified group was 336. The median IER ppm for the uncertified group was 3528, and the difference was statistically significant. This is the only Technology Segment where the 9100 certification indicated a statistically significant difference in IER. There was no difference with regard to OTD. The 9100 group had a median OTD of 95% (n = 68) and the uncertified group had a median OTD of 97% (n = 33). With regard to the use of SPC, only 1 respondent indicated "yes," and only 7 indicated "no." Again, due to small sample sizes (and both groups having a very low median IER), statistical comparisons could not be made. With regard to NADCAP, there was no statistical difference between the "yes" and "no" groups. The "yes" group had a IER median of 1017 (n = 24) and the "no" group had an IER median of 249 (n = 33). The OTD for "yes" group was 93% (n = 24) and for the "no" group was 97% (n = 33). 12. YEARLY COMPARISONS BY TECHNOLOGY SEGMENT TS1 IER With regard to TS1 IER, 2014 is the same as 2010 and but statistically better than IER for TS1 was unusually bad, however.

19 TS1 IER ppm Page 19 of 27 TS1 IER by Year with 95% Confidence Intervals for Medians n=89 n=74 Year n=121 n=487 TS1 OTD For TS1 OTD, 2014 is better than all previous years.

20 TS1 OTD% Page 20 of 27 TS1 OTD by Year with 95% Confidence Intervals n=89 n=83 Year n=115 n=487 TS2 IER With regard to TS2 IER, 2014 is better than 2010 and and 2014 are statistically the same.

21 TS2 OTD TS2 IER Page 21 of 27 TS2 IER by Year with 95% Confidence Intervals n=50 n=44 Year n=45 n=260 TS2 OTD With regard to TS2 OTD, 2014 is the same as 2013 and is better than TS2 OTD by Year with 95% Confidence Intervals n=50 n=54 Year n=42 n=263

22 TS3 IER Page 22 of 27 TS3 IER The TS3 IER for 2014 is borderline statistically worse than 2013 and The result was almost statistically significant at the 0.05 level, but not quite is statistically the same as TS3 IER with 95% Confidence Intervals for Medians n=48 n=49 Year n=56 n=147 TS3 OTD The TS3 OTD for 2014 is statistically the same as for all previous years. The TS3 OTD Medians and sample sizes for TS3 for each of the years were as follows: TS4 IER 2010: OTD Median = 79%, n = : OTD Median = 85%, n = : OTD Median = 85%, n = : OTD Median = 85%, n = 147. The TS4 IER for 2014 is the same as for previous years. The IER Medians and sample sizes for TS4 for each of the years were as follows: 2010: no data 2012: IER Median = 293 ppm, n = : IER Median = 429 ppm, n = : IER Median = 609 ppm, n = 103.

23 TS4 OTD Page 23 of 27 TS4 OTD The TS4 OTD for 2014 is the same as 2013, but better than There is no data for TS4 OTD with 95% Confidence Intervals for Medians n=39 n=32 n= SAMPLING STRATEGY The previous years' sampling strategy (2010, 2012, 2013) called for "random sampling" from the supply base. However, depending on the objective, this may or may not be the best choice. Suppose that 80% of supplied parts originate from 20% of the suppliers (call these suppliers Type A). Then, if we really want to estimate the typical experience of the customer (with regard to deliveries), the sample should be weighted heavily toward selecting the Type A suppliers. In fact, perhaps 80% of the sampled suppliers should be the Type A suppliers. The random sampling method gives a supplier who comprises 0.1% of supplied products the same chance of landing in the sample as a supplier who comprises 2% of supplied products (20 times larger). So, the actual experience being estimated is weighted heavily on the suppliers who do not contribute much to the customer/supplier delivery experience.

24 Page 24 of 27 Another problem with random sampling is that not all Technology Segments, Certifications, and Delivery Volume levels may be appropriately represented. For example, less than 13% of the polled suppliers indicated TS4, yet about half were from TS1. Similarly, less than 13% of polled suppliers indicated documenting SPC--which makes the sample size in that group small. If the IAQG wants to make inferences about the differences among Technology Segments, it may have issues due to insufficient information in some segments. To counteract these types of issues, the IAQG could consider "stratified sampling," and we could assist with the details. In the 2014 sample, the suppliers were supposed to be selected as a function of size. That is, the suppliers that contribute more supplied components were more likely to be selected for the sample than suppliers who contribute less to overall supplied components. This sampling strategy provides a better indication of whether overall performance is (or is not) improving with time. 14. SAMPLE SIZE The appropriate sample size for comparing data from various Years, Technology Segments, Certification Status, Number of Deliveries, etc. depends on several inputs: the degree of variation the risks of making erroneous conclusions the amount of difference (between groups) that is considered practically significant with regard to IER and OTD We could work with IAQG personnel to identify proper sample sizes to achieve IAQG objectives. In 2014, only about 5% of the supply base was sampled. Furthermore, the sampled suppliers often left potential predictors "blank" or indicated "unknown." Thus, of the 1084 respondents, the actual sample sizes were less due to non-responses. For example, when asked about the NADCAP accreditation, 39% indicated "unknown." For SPC use, 71% of responses were "unknown." 15. PERFORMANCE INDICATORS According to the website the Mission of the IAQG is to "achieve significant performance improvements in Quality, Delivery, and consequently Cost, on all products and services throughout the value stream: Through the establishment of effective prevention oriented practices and processes

25 Page 25 of 27 By standardizing Requirements, providing Process Guidelines and spreading Best Practices By introducing a Culture of Quality as early as possible in the value stream thus reducing the cost of poor quality Through establishing and maintaining dynamic cooperation between international Aviation, Space and Defense companies" With this in mind, there are other performance indicators that may be more indicative of quality and costs. We should consider some of these for future surveys. Keep in mind that any company can have a low IER, since an excellent inspection system will detect non-conforming product. So essentially, IER can be low as a result of costly inspection. However, as Dr. W. Edwards Deming states on p. 29 of his book Out of the Crisis, "quality comes not from inspection, but from improvement of the production process." With that in mind, perhaps other indicators of quality performance could be: First Pass Yield (FPY) Proper use of Statistical Process Control (SPC) Proper Process Capability Assessment and Actual Capability Estimates Evidence of adequate component or system reliability Internal Scrap Rates Internal Rework Rates Proper Measurement Systems Assessment, to ensure critical characteristics are adequately assessed Assessment of Standard Deviation and a commitment to reduce it for critical properties and features Use of Optimization methods to ensure optimal product and process performance Use of Statistical Methods (e.g. to understand differences between machines, cavities, part locations, spindles, nests, etc. -- and actions to correct for differences) Proper application of Designed Experiments and the mathematical models they produce for optimization Proper use of Reliability Methods to ensure that components perform adequately in application for their entire useful/intended life/application On another note, not all defects are equal. Some defects (or "escapes") are serious, since they could result in catastrophic or fatal consequences. Other defects result in inconveniences or are simply cosmetic issues. It may be too complex, but if not, perhaps defects and/or escapes could be weighted, depending on their severity. These are only a few suggestions, but perhaps the IAQG may want to supplement the data collected with one of more additional statistics.

26 Page 26 of 27 FINAL REMARKS: We used Minitab17 statistical software for the analysis and graphics, and we have extensive computer output to substantiate the stated results. Any or all supporting analyses, graphics, spreadsheets, etc. will be provided upon request. It has been assumed that the reported IER and OTD values were truthful, however, it is possible that some supplier data was not entirely thorough or accurate. Please do not hesitate to contact me if you would like any additional information or have any questions. Kind Regards, Allise Wachs, Ph.D. Integral Concepts, Inc.

27 Page 27 of 27 APPENDIX SUMMARY IER All Manufactured parts Subsystems and equipments Raw materials Hardware and technical products OTD All Manufactured parts Subsystems and equipments Raw materials Hardware and technical products Statistical improvement counts Statistical degradation counts No statistical change counts Weird Degradation IER 9100 cert No cert SPC no SPC NADCAP no NADCAP All Manufactured parts Subsystems and equipments Raw materials Hardware and technical products Significant Improvement of overall OTD No improvement at all for 5 years - worst result among all TS Overall slow but sustained improvement of our supply chain over the 5 last years SPC confirmed highly significant for better IER OTD 9100 cert No cert SPC no SPC NADCAP no NADCAP All Manufactured parts Subsystems and equipments Raw materials Hardware and technical products Legend: Small sample size for comparison Data does not exist Baseline Improvement Even more improvement overall adverse effect of certification on OTD