Risk analysis of parametric cost estimates within a concurrent engineering environment

Transcription

1 Risk analysis of parametric cost estimates within a concurrent engineering environment Rajkumar Roy, Sara Forsberg, Sara Kelvesjo, Christopher Rush Department of Enterprise Integration, SIMS, Cranfield University, Cranfield, Bedford, MK43 OAL, United Kingdom. Tel: 44 (0) r.roy@cranfield.ac.uk or c.rush@cranfield.ac.uk Abstract In industries where project time scales are long and the investment capital required is high, it is essential to minimise the risk involved with the development of a new product. Aerospace manufacturers are an excellent example of this kind of industry. Parametric estimating is a method used for predicting cost based on historical relationships between cost and one or more predictor variables. This method is commonly used during the conceptual stages of design where potential risk is at its highest. There is an increasing trend within parametric cost estimating to combine parametric analysis with statistical risk analysis methods. All cost estimates have a degree of uncertainty associated with them. The objective of a cost risk analysis is to predict the amount of uncertainties involved during the estimate of future projects. This paper presents the results of a risk analysis study conducted on a recently developed set of cost estimating relationships (CERs). The first case study considers the risks involved within a parametric cost estimate. This particular cost risk methodology can be used when only conceptual information is available. The main risk involved is related to the independent variable and its probability of change during the design process. The second case study provides the estimator with a possible range of costs for designing a part. This method can be used when there is little information available concerning a part. 1. Introduction Within the aerospace industry, where project time scales are long and the capital required is very high, it is essential to minimise the risks involved, especially during the early stages of a project s lifecycle. The objective of a cost risk analysis is to predict the amount of uncertainties involved in the cost estimate of future projects. There will always be uncertainties, i.e. risks, involved in a project. If these uncertainties can be identified and quantified, effort can be made to successfully deal with the impact of them occurring. Risk analysis is a very broad term, meaning the study of any situation, which is controlled in one way or another by uncertainty. The outcome of a risk analysis study may help to make extreme yes or no decisions, but it can also show all solutions between these extremes. By looking at the uncertain variables within a situation, a risk analysis can show which ones have the most effect on the solution, and pinpoint where most effort should be targeted. The risk analysis makes sure that uncertainty within the variable can be accounted for before committing the project. Therefore, the outcome of the analysis can be used as a decision tool for the designer, that is, if the designer understands the risks involved with certain cost drivers, he can choose a different approach to lower the risk. Thus, when using risk assessment and risk analysis it ensures that the consequences of risks to a programme cost and schedule are understood and taken account of for the commercial bid on programme price and duration [1]. This paper begins by highlighting the need for cost risk reduction initiatives within a concurrent engineering environment. It then examines different aspects of parametric costing and risk analysis with respect to their use for cost risk analysis. Section four describes the first of two risk assessment methodologies that can be used with a parametric cost estimate. Section five demonstrates how to use this methodology by applying it to a recently produced cost estimating relationship (CER). Section six describes the second risk analysis methodology and section seven demonstrates its practical application. The results of both methodologies are based on a CER that was previously developed within a European aerospace company [2, 3, 4]. In summary, the risk analysis methods examine whether the independent variables, or cost drivers, included in the CER change as the product matures through the design process. The results demonstrate how changes affect the accuracy of the cost estimate. The paper concludes the main findings of this research in section eight.

2 2. Aerospace concurrent engineering environment The company sponsoring this research embraces a concurrent engineering philosophy. Figure 1 illustrates a typical structure of an integrated project team by a family tree type representation. The tree illustrates the team members operating within the Technical and Engineering activities on the project under scrutiny [4]. Structural Mat Test Technical Task Leader Mech. Design Elec. Design Structural Tests Project IPT Manager Prod. Plan. Engineering Task Leader Assembly Detail Electrical N.C. Prog. Support Group Tool Design Tool Manf. Figure 1. Typical representation of IPT members Figure 2 represents a diagrammatic form of the IPT environment. The company recognised that new cost estimating relationships were required to reflect the activities of their concurrent engineering environment. The CER used for the risk assessment in this paper was developed around the activities within phase D. The reasons behind this were that data was available for the study within this area [13]. A, B & C D E IPT Technical Activities: 1. Mech. Design 2. Structure & Mass 3. Electrical Design 4. General Systems 5. Avionics Systems 6. Airworthiness 7. Structural Test 8. Material Test Engineering Activities: 1. Tool Design 2. Tool Manufacture 3. NC Programme 4. Electrical 5. Detailed 6. Assembly 7. Production Plan. Cradle to Grave Phases Tech. activities activities 1 st Article Production Service Figure 2. Diagram of Project IPD Environment 2.1. Risk analysis within a concurrent engineering environment The developed CER was designed for use during the early stages of design to predict the future effort of a new product development. The design of a new product goes through many phases. From the concept stage a SET (simultaneous engineering team) produces a project plan. From this plan initial schemes are produced until a mature concept is ready. The designs then pass through a freeze gate to prevent any further changes. Components are then separated from the Scheme and modelled on CATIA. These models then pass through a freeze gate before going on to the 2D stage (see Figure 3). Use CER in year 1 Concept Specs. Past Cases S. E. T. Project Plan Initial Schemes Initial Schemes To predict Initial Concept Phase CER 3D Freeze Concept Freeze Mature Concept 3D model Figure 3: Design activities Accumulated Engineering activities over a period of Years A B C D E, F, G F O C U S 2D Detail Activity Eng Freeze 2D Freeze 1 st Article An optimised concurrent engineering environment provides an opportunity to substantially reduce the total risk of a project. Because, integrated product teams (IPTs) containing members of various skilled disciplines, enable a simultaneous contribution to an early product development and definition. Therefore, within a fully integrated product development (IPD) cycle, multidisciplinary teams working together reduce the likelihood of product failure by avoiding costly alterations later in the design process. However, up to 70-80% of a product cost is committed during the concept phase of product development [5, 6, 7, 8]. Making a poor decision at this stage can be extremely costly. This is because product modifications and process alterations are more expensive the later they occur in the development cycle. Since the developed CER is intended to predict the cost of a product 20 years henceforth the sponsoring company wanted to understand the level of risk involved with the CER result. The remainder of this paper describes the process developed for this purpose.

3 3. Related research The following discussion captures different aspects of parametric costing and risk analysis within the conceptual stages of project development. This section provides fundamental knowledge concerning the tools and techniques currently used within the area of costing and risk analysis Parametric costing Parametric cost estimating (PCE) is a technique commonly used to estimate the cost of future systems. It provides a technique for predicting cost based on historical relationships between cost and one or more predictor variables i.e. cost estimating relationships (CERs). The method uses a statistical approach, and is commonly used during the conceptual design stages [9]. Mileham et al. [8] state that there is a growing need within the concurrent engineering environment to provide the designer with a simple, accurate method of estimating product costs during the conceptual stage of design. Their developed methodology is based on the basic information available to the designer during the conceptual design stage, and a set of data converters, which are used to calculate the values of cost-driving parameters. Parametric estimating can be used throughout the product life cycle. Both industry and Government accept the techniques. Many authors commend its usefulness [5, 6, 8, 10]. However, PCE does have its limitations, for example, CERs are sometimes too simplistic to forecast costs. If ill considered a CER could provide a completely misleading result. A broad outline of the CER development process is described below CER development process CERs can range from simple rules of thumb to complex relationships involving multiple variables. The principal function of CERs is to provide equations or graphs that summarise historical cost or resource data in a manner that will allow the equations or graphs to be used to estimate a future cost [5]. A general methodology for developing CERs includes activities such as data collection, testing a CERs logic, statistical analysis, CER significance tests and validation. Figure 4 illustrates this sequential process. The collection of data is often a very critical and timeconsuming activity. Typically more effort is devoted to assembling a quality database than to any other task in the CER development process. It is often said that the estimate is only as good as the data input to the cost model; therefore it is essential to collect accurate data. Data Collection Testing of a CERs logic Statistical Analysis Figure 4. A general process of developing CERs After a database is developed the first step is to hypothesise and then test the mathematical form of the CER. The analyst must determine and test a proposed CER, in order to determine its logic. The work involves discussions with engineers to identify potential cost driving variables, scrutiny of the technical and cost proposals, and identification of cost relationships. Only with an understanding of estimating requirements can an analyst attempt to hypothesise a forecasting model necessary to develop a CER. In order to test and validate a CER statistical analysis is used. Multiple regression is the most common method used to test hypotheses [11]. Although widely accepted PCE is primarily based on statistical assumptions concerning the cost driver relationships to cost, and therefore, estimators should not completely rely upon statistical analysis techniques. Hypotheses, common sense and engineering knowledge should come first, and then the relationship should be tested with statistical analysis. Because of the identified limitations within PCE and CERs there is an increasing trend with parametric cost estimators to combine the statistical techniques of parametric analysis with statistical risk analysis methods. Parametric estimating offers the cost analyst the advantage of being able to quantify the risk. This helps to restore confidence within the results and improves decision-making Risk Analysis Testing of the CERs Significance Validation of the CER Edmonds [1] discusses the advantages of using risk management within the aerospace industry. Stating that the use of risk assessment and risk analysis ensures that the consequences of risks to programme cost and schedule are understood provided they are taken account of during the commercial bid of a programmes price, and duration. A cost risk analysis generates a range of costs for a project, and assigns a probability level to each cost value within the range. The introduction of risk assessment and risk analysis ensures that the goals of the producer and consumer

4 materialise and that they both benefit. It provides confidence concerning the final product and identifies actions needed to keep cost and schedule on target. There are five key steps to follow in the risk management process [12]. Figure 5 illustrates this process more clearly. Identify Assess Feedback Analyse Reduce One of the most important benefits of using risk assessment is to generate a distribution/range of costs i.e. to move away from single point estimating, since a range of costs are much easier to estimate than a single cost [13]. Furthermore, once a risk analysis has been conducted the analyst can consider ways to reduce the risk e.g. by avoidance, deflection or contingency and then plan accordingly to control the reduction process. Statistical theories are well established and there are many software packages available to perform such calculations. However, for risk assessment the situation is very different and software is not so readily available. One computer model that can, and is used for the risk analysis of cost estimating is named SAM [14] (Stochastic Aggregation Model). SAM is a Monte Carlo simulation program designed to help the cost analyst quantify the uncertainty associated with a parametric cost estimate. The areas of cost risk that SAM evaluates are: cost estimating relationships (CER); independent variable uncertainty; complexity factor uncertainty, and; CER statistical uncertainty. Control Figure 5: Risk Management Process Crossland et al. [15] recognised that in the early stages of a design project, it was necessary to evaluate the design to determine its feasibility in order to focus attention with particular areas of the design. They developed a tool-set called RiTo (RIsk TOol) to fulfil this purpose. RiTo, is an objective-oriented risk modelling tool that was developed to support the development of design from the earliest stage by building models for risk assessment. By combining risk analysis with the normal pricing process, the estimator gets a direct measure of the risk at the same time the estimate is formed, and therefore can allow for a contingency value to be quantified. This contingency provides a better understanding of the correlation between items, which can have large combined effect on the overall distribution [16]. The remainder of this paper discusses two different approaches to the cost risk analysis process. Each approach is tested on case studies in order to validate their applicability. 4. Cost risk methodology one 4.1. Identification The first step in handling risks is identification. This is readily available from the results of the cost estimating relationship. The risk consists of the independent variables; included in the CERs (see below), and their probability of changing throughout the process. The independent variables, also called cost drivers, are selected through statistical analysis and form the basis of the CER. Y = C 0 + C 1 (Mass) + C 2 (Surface related) Where: Y = The dependent variable, time; C 0 = The constant; C 1 = The constant multiplied with the value of mass, and; C 2 = The constant multiplied with the surface related. Because the independent variables are prone to change as the product matures through the design process the accuracy of the cost estimate is affected. The higher the probability of the change occurring, and its impact, the less accurate the cost estimate will be Assessment Having identified possible sources of risk, included in the cost estimate, the analyst then needs to calculate their impact on the cost. This is the risk assessment. The risk can be defined as: Risk = p * c, Where: p is the probability of the event occurring, and; c is the impact of the risk on the estimate. This means that both the probability of the risk occurring, and the impacts are assessed. The quantification of the cost risks can be made through a probability distribution. One way of representing this is through a triangular distribution. To achieve this the minimum, most likely, and maximum cost are required, see Figure 6 below.

5 5. Case Study One Minimum Most Likely Maximum Figure 6: Triangular distribution The values of the impact are then taken from the coefficient of the independent variable of the CER and multiplied with the values gathered from the risk assessment. As discussed earlier one of the most important benefits of using risk assessment is to generate a distribution of costs, and move away from single point estimating Analysis After the risk assessment is performed, the analysis can be carried through. This is completed using simulation, such as Monte Carlo or Latin Hypercube. The outcome of the risk analysis provides the estimator with a range of costs, instead of a single one, from the CERs. In this way, it can be assumed that the cost will not exceed a specific value, with a certain probability; normally an 85 % probability is used. This is presented using a cumulative probability curve, the S curve, see Figure 7. % Cumulative Probability A case study was conducted where the risks involved from a parametric cost estimate were considered The estimate, a Cost Estimating Relationship (CER), was developed to capture the design time of producing a specific part [2, 3, 4]. This cost risk analysis methodology can be used during the conceptual design stage when there is limited information and uncertainty concerning the independent variable. That is, there is a probability of the independent variable values changing during the design process Identification In this study, the risk incorporates the independent variables from the CER i.e. mass and surface related. The value of these independent variables can change during the design process, therefore, they are considered as risks Assessment With the assistance of expert opinions an assessment of the risks, included from the CER, were conducted. The identification and evaluation of the two independent variables are illustrated in Table 1. Mass risk Surface related risk Description Risk An increase in the mass (within the time span that the estimate covers). Risk A change of the product whether it is surface related or not. Probability 50% 0% MIN 5% of the mass - Most Likely 20% of the mass - MAX 50% of the mass - Table 1: The identification and assessment of the different risks involved in the cost estimate Figure 7: A cumulative probability curve 4.4. Mitigation Time If the risk involved is shown to be very high, a mitigation plan should be made. The mitigation plan should include discussion with experts concerning the area under consideration and estimators who have experience in risk management. According to the experts, the surface related variable did not have any probability of changing during the design process. This meant that the variable was no longer considered as a risk, and it was therefore excluded from further analysis. The percentages of the likely changes of the mass figure were gathered from the experts. The independent variable, mass, and the impact, c, were then distributed using the triangular distribution. Since there was a large range within the mass of the parts that the CER was based upon, the impacts were grouped into five different weight categories. The averages of these weight categories were then multiplied with the risk percentages and the coefficient of the CER, see example below. This sum was then used as the impact of the risk described in a triangular distribution.

6 (min % of change in mass, most likely % of change in mass, max % of change in mass) *(average weight of the specific category) *(coefficient of the variable from the CER) = Triangular distribution (impact) of the cost estimate. Worked example: (5%, 20%, 50%)*(800 gram)*0.05 = (2 hours, 8 hours, 20 hours) The probability of the risk occurring can be shown in a discrete distribution. This type of distribution is used when there is a known amount of outcomes, i.e. the risk will occur or not. Discrete ({0,1}, 50%, 50%}) The probability of the risk occurring and the impact of the risk are then multiplied to compute the actual risk. This is achieved by multiplying the discrete distribution and the triangular distribution together Analysis During the identification and evaluation of the risks a tool named PREDICT! was used to perform the analysis. This tool is generic and not specifically designed for cost risk analysis. The analysis was made using a Monte Carlo simulation Results The output produced from the analysis is illustrated using a cumulative probability curve. This plot shows that there is a probability of 80 % that the extra design time, if mass increase during the design process, will not exceed 10 hours. See Figure 8 below: % 6. Risk analysis methodology two To predict the range of a CER, when limited information is available about the product i.e. a new concept/idea, another cost risk analysis is required. This provides the analyst with a technique that can predict the maximum outcome of the cost estimate, with a specific probability Identification As previously mentioned, the risks involved that can cause changes within the CER range, are the independent variables. These independent variables are readily available from the CER since they form the basis of the estimate Assessment The probability distributions of the independent variables are also readily available from the data that was used for the development of the CER. These data can be distributed as a triangular distribution with a minimum, most likely and maximum value and used for the assessment of the probability, see Figure 6 above Analysis To perform the risk analysis, a software package can be used. The input data is taken from the identification and evaluation of the risks, or the independent variables included within the estimate. Because probability distribution data are readily available from CERs, parametric costing is sometimes combined with cost risk analysis techniques involving Monte Carlo simulation. The simulation performs several thousand iterations to form a frequency distribution of cost [5] Results The outcome of this type of cost risk analysis is a range of costs for the total CER. This type of analysis is used when there is only a limited amount of information available. The analyst can say to a specific probability, that the cost of a particular type of part, used for the cost estimate, will have a certain cost range. This can be extremely useful for early estimates, when little or no information about a part is available. Figure 8: Cumulative probability curve Time 7. Case Study Two The same CER, used within case study one, was used for this case study. This cost risk analysis provides a range

7 of total costs, to a certain probability, for designing a specific part Identification The CER used two independent variables, mass and surface related that could cause changes within the CERs range. From the data set used for the CER development, the variation of the independent variables, mass and surface related, were identified Assessment The range of mass, which was readily available from the data set, was assessed for the analysis. A minimum, most likely and maximum weight was captured. The other independent variable, surface related had only two possible outcomes, surface related or not Analysis An analysis was conducted on the independent variable mass. A triangular distribution, which included the impact mass had on the CER, was used in the simulation tool PREDICT Result The impact that mass had on the CER is illustrated in the graph, Figure 9. The graph illustrates with an 80% certainty the impact mass has on the CER will not exceed 67 hours. % Total time = C0 + [C1 (mass)+maximum added time if the weight change] + C2 (surface related) Thus, the total time will not exceed 213 hours, with the probability 80%. This is the absolute maximum value that the estimate can be. The second independent variable is chosen to find the maximum impact of the total cost. 8. Discussion and Conclusions This paper has presented two different methodologies for risk analysis of CERs within a concurrent engineering environment. The risk identification and assessment is very much dependent on the environment. When conceptual information is available and a cost estimate is completed using a CER, a risk analysis like the one used in case study one, will provide the analyst with the extra amount of time the design process will take if the independent variable changes, within a certain probability. In some cases when the conceptual information is not totally defined a risk analysis can be very useful for the estimator. The analysis will indicate the upper and lower boundary of cost that a part will obtain as it is being designed. These case studies are based on a CER that was developed for a specific type of part; a small sample size was used for the CER development. Therefore, the results from these case studies can only be used for the same type of part. Nevertheless, the results indicate the possibility of using the two approaches as predictive tools for quantifying the risks involved when using CERs within a concurrent engineering environment. Acknowledgements The authors would like to thank Ian Taylor of BAE SYSTEMS who helped to develop and supervise this research with Cranfield University. Figure 9: Cumulative probability plot Time This result is then summarised with the outcome from the CER. The maximum occurring time is added to the independent variable mass: References [1] EDMONDS, R. J. A case study illustrating the risk assessment and risk analysis process at the bid phase of a project. British Aerospace Defence Limited, Dynamics division. Ch. 2. [2] ROY R., BENDALL D., TAYLOR J. P., JONES P., MADARIAGA A. P., CROSSLAND J., HAMEL J., TAYLOR I. Development of airframe engineering

8 CERs for military aerostructures. MSc group project, Cranfield University, UK, [3] ROY R., BENDALL D., TAYLOR J.P., JONES P., MADARIAGA A. P., CROSSLAND J., HAMEL J., TAYLOR I. M. Development of airframe engineering CERs for military aerostructures. Second World Manufacturing Congress (WMC'99), Durham (UK), 27-30th Sep., [4] ROY R., BENDALL D., TAYLOR J.P., JONES P., MADARIAGA A. P., CROSSLAND J., HAMEL J., TAYLOR I. M. Identifying and capturing the qualitative cost drivers within a concurrent engineering environment. Advances in Concurrent Engineering, Chawdhry, P.K., Ghodous, P., Vandorpe, D. (Eds), Technomic Publishing Co. Inc., Pennsylvania (USA), pp , [5] STEWART, R., WYSKIDSA, R., JOHANNES, J., Cost Estimator's Reference Manual, 2nd ed., Wiley Interscience, [6] DEPARTMENT OF DEFENCE. Parametric Estimating Handbook, 2nd Ed., DoD, [7] TAYLOR, I. M. Cost engineering-a feature based approach. In: 85th Meeting of the AGARD Structures and Material Panel, Aalborg, Denmark, October 13-14, 14:1-9, [8] MILEHAM, R. A., CURRIE, C. G., MILES, A. W., BRADFORD, D. T. A parametric approach to cost estimating at the conceptual stage of design. Journal of Engineering Design, 4 (2): pp , [9] RUSH, C. & ROY, R. Analysis of cost estimating processes used within a concurrent engineering environment throughout a product life cycle. Proceedings, 7 th International Conference on Concurrent Engineering, University Lyon 1, France, July th, [10] PUGH, P. Working top-down: cost estimating before development begins. Journal of Aerospace Engineering, Part G, Vol. 206, pp , [11] NORUSIS, M.J. SPSS 8.0 A guide to data analysis. Prentece-Hall, Inc, New Jersey, 1998 [12] TURNER, R: J. The handbook of project-based management. McGraw-Hill International (UK) Limited, [13] FORSBERG, S., KELVESJÖ, P. S. CER development for airframe engineering. Masters Thesis, Cranfield University, UK, [14] HAMAKER, J. W. SAM user Manual. Huntsville, AL, Cited in: Stewart, R. D., Wyskida, R. M., Johannes, J. D. (ed). Cost estimator s reference manual, ch 8, [15] CROSSLAND, R., SIMS WILLIAMS, J. H., MCMAHON, C. A. An object - oriented design model incorporating uncertainty for early risk assessment. In: International Computers in Engineering Conference, Boston, MA, [16] HULL, K. AEPS/ETG, Ministry of Defence, Procurement Executive. Risk analysis techniques in defence procurement. (unpublished), 1991.