Predicting Over Target Baselines (OTBs) 1

Transcription

1 2010, Issue 4 The Magazine of the Project Management Institute s College of Performance Management Predicting Over Target Baselines (OTBs) 1 By Kristine Thickstun, MBA, MS, and Dr. Edward D. White Stick OUT in a crowd: Advertise in the Measurable News is accepting sponsorship for future issues. With sponsorship, you receive: 4-color, 1/3-page ad on front cover 4-color, full-page ad on back cover Priority publication of two articles from professionals in your organization Too good to be true? Find out more! Contact Managing Director Gaile Argiro ExecAdmin@pmi-cpm.org Abstract In 2009, Captain Trahan developed several Gompertz growth curves, which provide better estimates at completion (EACs) for over target baseline (OTB) contracts within the Air Force. We investigate whether we can predict when OTBs occur to utilize this method. While we are unable to develop a model to predict OTBs, our results indicate that OTBs tend to occur randomly, which leads us to question the implications of using OTBs. Background Cost estimators use a variety of methods to develop estimates at completion (EACs). The most commonly used method for calculating an EAC is an indexed-based approach. Using an indexed-based approach, analysts calculate the EAC by taking the sum of two items: 1) the actual cost of work performed (ACWP) and 2) the estimated cost of remaining work, which is the budget at completion (BAC) minus the budgeted cost of work performed (BCWP), divided by a performance index. This estimate assumes that future performance will be similar to past performance. Other methods for developing EACs include time series techniques, using simple performance factors, and a variety of regression techniques. Many researchers search for new and better techniques for developing EACs. In 1995, Dr. Christensen reviewed 25 EAC studies and concluded, The accuracy of regression-based models over index-based formulas has not been established, and The accuracy of index based formulas depends on the type of system and the stage and phase of the contract (Christensen, 1995). His research indicates that there is no best method for developing EACs for all contracts. Additional research in using Continued on p This article is an adaptation from Captain Thickstun s Masters thesis undertaken as part of a PMI CPM-sponsored research project. PMI CPM Research and Standards established a sponsored research program for EVM related research with the US Air Force Institute of Technology (AFIT) in See our VENDOR / SERVICES listing on page 16.

2 , Issue 4 continued from p. 1. Figure 1. Development Growth Model (Trahan, 2009) regression to develop EACs also proves that no single technique for developing an EAC works best for all situations and all types of contracts. These conclusions make a strong case for using specific forecasting methods that perform better under specific circumstances. One case in which a specific estimating method performs better in certain cases is the use of a Gompertz growth curve. In 2009, Captain Trahan investigated the use of these curves to model costs for Air Force acquisition contracts. The Air Force tends to incur costs for these contracts in an S-shaped manner; that is, a contract tends to incur costs slowly at the beginning, then costs rapidly accrue until they taper off at the end. Based on this observation, a growth curve would appear to be appropriate as it also exhibits an S shape. By fitting growth curves, Captain Trahan was able to develop growth functions for the contract s estimated growth in spending based on the percent of time complete. Figure 1 depicts one such growth curve. Using the growth functions, Captain Trahan was able to estimate the remaining amount of growth in spending for a particular contract. Based on this information, she could calculate the EAC for each contract by adding the actual costs incurred to the projected remaining costs, which incorporates the remaining growth in spending. To determine whether the Gompertz growth curves developed by Captain Trahan are better than the standard index based approaches, she 1) compared her EACs to the actual costs incurred and 2) compared the EACs using indexed-based approaches to the actual costs. Using the Mean Absolute Percentage Error (MAPE) statistic, which compares the average percentage of error for each technique, she discovered that the Gompertz growth curves did not perform better in most cases. However, she concluded, Our growth models present a better model than the popular index-based methods currently in use for estimating OTB contracts specifically, as these models outperformed the indexed-based approaches 71% of the time (Trahan, 2009). Captain Trahan s model is an excellent model to use for developing EACs for over target baseline (OTB) contracts or contracts with a high likelihood of becoming an OTB contract. However, we do not always know whether a contract will become an OTB contract in the future. What is an OTB? An OTB occurs when the contractor cannot complete the remaining amount of work within the original budget and the work scope has not changed (Cukr, 2001). According to the Defense Acquisition University (DAU) handbook on OTBs: An OTB is a contract budget base that was formally reprogrammed to include additional performance management budget and which therefore exceeds the contract target cost [And] ANSI/EIA defines it as a recovery plan, a new baseline for management when the original objectives cannot be met and new goals are needed for management purposes. (DAU, 2003) When an OTB is used, the program manager is recognizing a cost overrun. We identify an OTB when the total allocated budget (TAB) exceeds the contract budget base (CBB) as depicted in Figure 2. The difference between the TAB and CBB is the recognized cost overrun. Approximately 20% of all acquisition contracts in the DoD experienced cost overruns over the past 20 years (based on analysis of contracts in the DAES database). The average cost overrun for each contract experiencing an OTB is $321 million (BY09$). In the process of implementing an OTB, a new performance measurement baseline (PMB) is devel-

3 2010, Issue 4 11 oped and the cost and schedule variances are often set to zero. This allows program managers to obtain a clean slate with which to work. While this seems to make an OTB the preferred method for dealing with substantial cost overruns in defense acquisition programs, contractors do not always use an OTB. The OTB process is a lengthy 10-step process that can be costly and take many months to complete. Furthermore, any time spent on developing a new baseline for management purposes may delay progress made on the contract itself. Figure 2. EVM Contractual Price Components (DAU, 2009) Predicting OTBs To implement Captain Trahan s growth models, we need to know which contracts are currently OTB contracts and which contracts will become OTB contracts in the future. If we already know which contracts are already OTBs, we can simply apply her models to develop better EACs. However, if we do not know if a contract will become an OTB, we must predict whether it will become an OTB. If we know a contract is likely to become an OTB, we can use Captain Trahan s models to develop EACs. The models that we develop attempt to predict a dichotomous response with two possible outcomes: 1) the contract will become an OTB or 2) the contract will not become an OTB. To predict such an outcome, we utilize logistic regression models. In our models, we assign contracts a value of one for the outcome if it becomes an OTB and a value of zero for the outcome if it does not become an OTB. We fit logistic regression models in JMP to predict whether OTBs will occur for a given contract. The outcome for each of the logistic regression functions that we fit is a probability, which represents the likelihood that a contract will become an OTB. Figure 3 provides the typical functional form of a logistic regression equation. In this equation п(x) is the likelihood of a contract becoming an OTB, each x n is a predictor variable, and each Bn is the associated coefficient that we fit for each predictor variable in п(x) = e B 0 +B x + B x + +B x n n 1+e B 0 +B x +B x + +B x n n Figure 3: Logistic Regression Equation the model. In our models, an outcome of п(x) = 0.75 would indicate that the specific contract has a 75% chance of becoming an OTB contract. The data that we use to develop models to predict OTBs consists of acquisition contracts across the DoD. Using the Defense Acquisition Management Information Retrieval Database, we consider Defense Acquisition Executive Summary (DAES) data on major defense acquisition programs (MDAPs) and major automated information system (MAIS) programs. We include a variety of potential predictor variables in our model, such as contract characteristics, earned value management (EVM) metrics from the cost performance reports, production quantities, development quantities, average procurement unit costs, and threshold breaches (APB and Nunn- McCurdy). Additionally, for each program, we identify the program type (MDAP, MAIS), acquisition category, and commodity type. By including a vast array of potential predictor variables, we attempt to capture the variables that best explain whether an OTB occurs. Using the available data on DoD contracts, we build the best possible logistic regression models to predict OTBs. The models that we build could assist

4 , Issue 4 us in identifying OTBs and lead to the development of better EACs for identified OTB contracts. Furthermore, these models explain why OTBs occur. Since an OTB formally recognizes a cost overrun, these models also explain why cost overruns occur. Therefore, by building models to predict OTBs, we learn what variables have the greatest influence on the occurrence of cost overruns. Our initial attempt at building models to predict OTBs included development and production contracts; however, this process did not produce any statistically significant models. Therefore, we separate the models into two types: 1) development contracts and 2) production contracts. This is consistent with previous EAC studies where researchers developed separate models for each of these two types of contracts. Previous researchers modeled these contracts separately due to the inherent differences between the development and production phases of the acquisition cycle. For each set of contracts (development or production), we use a stepwise procedure in JMP to develop a series of logistic regression models. We also use a more subjective approach of adding variables to models that appear to be better predictor variables. Other EAC studies use this subjective approach to include common predictor variables that have the potential to be good predictor variables. In our study, we add variables that measure contract performance, such as the cost performance index (CPI). Once we have multiple models to choose from, we determine the best models to use based on three important criteria: 1. The model s overall significance, given by a p-value associated with the chi-square statistic of less than The model s r 2 (U), a measure of the proportion of the total uncertainty that is attributed to the model fit for a logistic regression model (JMP, 2009) (higher values are better). 3. The area under the receiver operating curve, which provides a measure of the model s ability to discriminate between those subjects who experience the outcome of interest versus those who do not (Hosmer and Lemeshow, 2000) (higher values are better). Tables 1 and 2 provide the best models for development and production contracts, along with their associated summary statistics. These models capture the reasons why contracts became OTB contracts in our sample. (It is important to note that the positive or negative signs of the coefficients for each predictor in JMP are the opposite of what they would be in the standard logistic regression equation presented in Figure 3.) We chose to analyze two models for each of the contract types to determine if slightly different sets of predictor variables performed differently in the validation step. As an example, suppose we would like to use the five-variable development model to predict whether a development contract will become an OTB. Hypothetically, suppose we have an Air Force contract for a fighter aircraft that is both behind schedule and over cost with an SPI and CPI each of We will assume that an APB breach has not occurred yet either. Then we would calculate the outcome of our model as follows: ( *1) + ( *0) + ( *1) + ( *0.90*0.90) + ( *0) п(x) = e ( *1) + ( *0) + ( *1) + ( *0.90*0.90) + ( *0) 1+e п(x) = e e п(x) = 0.87 Analysts would interpret this outcome as an 87% chance of the contract becoming an OTB in the future. Since the probability is greater than a 50% chance, we would predict that this contract would experience an OTB. The results of the development models indicate that Air Force, Navy, and fighter aircraft contracts are more likely to experience an OTB. Contracts with a low SPI*CPI, also known as the SCI, are more likely to experience an OTB. A low SPI*CPI occurs when the contract is behind schedule, over budget, or both. In the five-variable model, a contract that has not experienced an APB performance breach yet is likely to experience an OTB. In the sixvariable model, contracts with a high EAC and contracts with a low value for percentage complete are more likely to experience OTBs. That is, a contract that is in the early stages in terms of percent complete is more likely to experience an OTB. For production models, contracts that have a high BCWS and a low BCWP are more likely to experience

5 2010, Issue 4 13 Table 1. Development Model Table 2. Production Model an OTB. This means that contracts with a large amount of work scheduled (BCWS) and a small amount of work performed (BCWP) are more likely to experience an OTB. Contracts that have experienced a large change in the production quantity since the initial report are more likely to experience an OTB. A contract that has experienced an APB schedule breach is also more likely to experience an OTB. For the five-variable model, a contract with a large EAC is more likely to become an OTB. For the six-variable model, a contract with a low value for percentage complete (early on) and has experienced an APB performance breach is more likely to experience an OTB. Before concluding that these models can be applied to predict OTBs for other contracts, we must validate their predictive ability. Using data that we initially set aside (20% of our data set); we test the performance of the final four models. We calculate the logistic response for each of the entries in our validation set. Using a cutoff of 0.5, we predict an outcome greater than this value to be an OTB and an outcome less than this value will not be an OTB. We then compare our predicted values to the actual outcomes for each contract. The validation for the development models indicates that when we predict an OTB we are incorrect half of the time, which means our model is no better than a coin flip. Furthermore, our models often fail to predict an OTB when they occur. The validation results for the production models show that when we predict an OTB, we are accurate the majority of the time, however, we fail to predict most OTBs. That is, for production contracts, we are only predicting approximately 1 out of 100 OTBs. This predictive ability is not useful since our goal is to predict OTBs, which we rarely do with these models. Therefore, our validation results indicate that cost estimators cannot use these models to predict OTBs for other contracts. Before concluding that we cannot predict OTBs, we examine our assumptions and consider other approaches to building our models. First, we consider the time period for the occurrences of OTBs. By limiting our analysis to times when OTBs are more likely to occur, in terms of percent complete, we were unable to improve our models. Second, we consider only those instances where a program office formally recognized an OTB. Again, this attempt did not change our results. Finally, we searched for trends among the predictor variables for OTB contracts. This analysis provides no such common trends among the predictor variables we

6 , Issue 4 consider. These observations suggest that OTBs may occur randomly. Conclusions and Implications The initial purpose of this research was to develop a model to predict OTBs so that we can 1) use Captain Trahan s models to develop better EACs for OTB contracts and 2) predict OTB related cost overruns. However, by considering a wide range of predictor variables in multiple attempts, we are unable to validate any of our models that predict OTBs. Therefore, we are unable to use these models to predict OTBs in the future. The inability to develop models that predict OTBs indicates that the predictor variables do not explain whether an OTB will occur. This implies that there is no set of common characteristics representative of OTBs, both qualitatively and quantitatively and leads us to conclude that OTBs may occur randomly. When we revisit the OTB process, we recognize that the current DoD acquisition process does not require contractors to implement an OTB under any specific guidelines. In fact, contractors may conduct OTBs at their own discretion; therefore, contractors may not use the OTB process when it could be beneficial to them. Based on our analysis, two contracts may have the same characteristics and similar performance measures, but only one actually uses the OTB process. This leads us to the following question: what is the benefit of an OTB? Do we see better performance over time for an OTB contract? Does the CPI performance improve in the long term after an OTB? (A topic currently being examined by Major Dennis Jack at the Air Force Institute of Technology.) Is there less overall cost growth for an OTB contract? If we do see improved performance, we need to encourage or mandate using OTBs. However, if this discretionary use of OTBs results in little or no improvement, we should not devote the time and resources to establishing an OTB. References Christensen, D.S., R.D. Antolini, J.W. McKinney, and The Air Force Institute of Technology A Review of Estimate at Completion Research. Journal of Cost Analysis, 41:62. Cukr, A When is an Over Target Baseline (OTB) Necessary?, March. Defense Acquisition University Earned Value Management Gold Card. January. Defense Acquisition University Over Target Baseline and Over Target Schedule Handbook. Hosmer, David W. and Stanley Lemeshow Applied Logistic Regression. New York: John Wiley & Sons. JMP Version 8. (Academic) computer software Cary, NC: SAS Institute Inc. Trahan, E An Evaluation of Growth Models as Predictive Tools for Estimates at Completion (EAC). MS Thesis, AFIT/GFA/ENC/ Graduate School of Engineering and Management, Air Force Institute of Technology, Wright Patterson AFB OH. About the Authors Captain Kristine Thickstun graduated from the Air Force Institute of Technology in March of 2010 with an MS in cost analysis. She is currently working as a weapon system cost analyst at the Space and Missile Systems Center at Los Angeles Air Force Base, CA. Edward White, Ph.D., is an Associate Professor of Statistics within the Department of Mathematics and Statistics at the Air Force Institute of Technology. His research interests include cost analysis, regression modeling, design of experiments, growth curves, and biostatistics.