Analysis of evaluation time frames in project portfolio selection

Transcription

1 Mat Independent Research Projects in Applied Mathematics Analysis of evaluation time frames in project portfolio selection December 10 th, 2007 Helsinki University of Technology Department of Engineering Physics and Mathematics Tuomas Nummelin 62832W

2 Table of Contents 1. Introduction Research objectives Model The Project Portfolio Selection Problem Complete Information Incomplete Information Portfolio Selection Rules Time Frame Time, Accuracy and Price Simulation Model Example Background Story For A Situation Where Portfolio Selection Problem Can Occur Simulation Parameters The Expert Evaluations Joint Distributions Parameters Results Limitations of Simulation Summary and Conclusions References

3 1. Introduction Portfolio decision analysis is one of the major applications of decision analysis for business and governments. It is used to support investment strategies and allocation of resources. The purpose of portfolio selection problem is to provide a structure or a framework to valuate project proposals. Models or processes for a selection problem are mathematical constructions to evaluate probabilities of random events to bring consensus to a decision maker s (DM) decisions and thus reveal models of expected utility to the DM. Information produced by the results of a portfolio selection process can be valuable to the DM; though the value of process itself might be also very valuable, because the use of selection process provides a structural way to evaluate the selection in hand. The rationale behind the constructing analytical decision models in portfolio selection is that they are expected to improve quality of the decision. Even though the value of the process in certain decision situations might be hard to measure due to various reasons, the predominant of them being that uncertainty involved even in the best decisions may yield low results. Also, because the performance of unselected alternatives is not usually measured, the comparison between the alternative and those not selected cannot be done. To avoid these problems ex post analysis of the value, Nickerson and Boyd (1980) offer a normative approach to the valuation of the decision analysis. They consider the use of decision of model as an additional source of information to the DM. The value of decision analysis is usually defined e.g. Watson and Brown (1978). Approaches like the ones mentioned offer a way to compare benefits from analytical processes to costs of analysis. Analytical decision methods are usually applied to resource allocation decisions that organizations and individuals make Kleinmuntz (2007). Typically organizations have more good opportunities or project proposals than they can pursue due the limited resources. Typically this kind of allocation involves significant capital investments in industrial organizations and in the public sector (e.g. Golani et al. 1981). Various tools and analytical decision approaches have been developed to help the DM to choose which alternatives would be good. In this kind of a decision situation, there can be hundreds of alternative project proposals and time constraints, so the DM, in addition to her own opinion, has to rely on some additional sources of information; often to some other people who have special knowledge of the project proposals, i.e. expert evaluations are ordered. However, this information can be inaccurate or incomplete. The DM is thus forced to make a decision in an uncertain environment. It can nonetheless be assumed that the DM has the knowledge to combine available information in an efficient way to maximize its value to her. The value of portfolio selecting methods has been widely studied in both aspect evaluation practicalities with companies Clemen and Kwit (2001) and in case studies Keisler(2007); both approaches aiming to evaluate the added value of the selection process. The simulation framework approach in the selection problem, in which the projects do not necessary have a direct monetary value (e.g. R&D project proposals), has been examined by Nissinen (2007). 3

4 1.1 Research objectives The objective of this study is to examine portfolio selection processes in a situation where the arrival of expert evaluations is uncertain. Reason for uncertainty is time, i.e. is the evaluation assessment ready and delivered to the DM before she has to make a decision. The first objective is to form a model for this kind of time dependent situations and to introduce that model to a simulation model. The second objective is to analyze what kind of phenomena occurs in time dependent portfolio selection process. 2. Model The purpose of a time frame model is to expand the project portfolio selection problem by means of recognizing that the time of arrival of expert evaluations may be uncertain. We will briefly, but formally define this portfolio selection problem in Section 2.1 by introducing the essential variables and decision criteria. A brief discussion about the optimization procedure is also presented. The original time independent model is introduced in reference [1] The Project Portfolio Selection Problem Complete Information Let us denote the set of project proposals as. We assume that the benefit, i.e. value that accrues to a decision maker (DM), is measured with a single performance score, which is stored in a value vector. Project portfolio is a subset of all project proposals. We assume the values of the projects to be independent of each other so that whether one project is started or not, the values of any other project are not affected. Thus we can calculate an overall value for the portfolio as the sum of its constituent projects values. We denote the aggregate value of the portfolio with, with the assumption of additivity (see e.g Golani et al 1981 and Chien 2002). Thus (1) where is a mapping so that if and only if. Usually there are some resource constraints that limit the feasible portfolios. For simplicity, we assume a single resource model in which a project consumes a certain amount of a limited resource (e.g. money) that is only available for units. Similar to delivered values by the project proposals, costs, stored in a cost vector, are assumed to be independent of each other. The cost of a portfolio can be calculated as the sum of the associated costs of projects. Satisfying the resource constraints, we obtain 4

5 (2) where the inequality holds component-wise. The DM can use these definitions to form a linear optimization problem (LP problem), with which she can identify the most preferred, 'optimal', portfolio as a feasible portfolio, that maximizes the aggregate value of its constituent projects. The problem can now be solved using proper algorithms for LP problems e.g. SIMPLEX-method [2]. (3) We denote the optimal portfolio given by (equation (3)) with Incomplete Information In real world situations a DM may not knows the exact model parameters, i.e. the cost and value vectors. However, it is common that the cost of a project proposal is more readily assessed than the benefits or the value that it would deliver, if the project was funded. This is true in many real life situations, e.g. the costs of new machinery can be estimated with some degree of accuracy, but the value, that the new machine will aggregate, incorporates great uncertainty. We assume here that the DM knows the costs of the projects. Although the real values of the project proposals remain unknown to the DM, we have to assume that the DM has some way of evaluating their value and set a metric rank order to the project proposals. If not, the portfolio selection is reduced to a random selection process. We introduce a new variable to denote the belief the DM has about the value of the project proposal. We assume that the single criterion behind the DM s decision is this belief, i.e. it incorporates all the information the DM has about the project proposal [3]. The DM may be able to evaluate the value of this parameter herself or she may need outside help in evaluating it. The DM may consult one or more outside experts to evaluate the value. Let us assume that the evaluations of the project proposals require work, i.e. it consumes available resources. We denote these evaluation costs with, which in the set of feasible portfolios must satisfy (4) 5

6 Instead of solving the optimal portfolio LP problem based on the actual outcomes of projects as in equation (3), the DM has to make her choice based on her belief vector. The optimization problem is very similar to the one presented in equation (3). (5) Similar to the notation used for the problem with complete information (3), we denote the optimal portfolio given by equation (5) with and. Also. The portfolio is chosen based on the ex ante evaluation information of the project proposals available to the DM. Therefore, there is no need for it to contain the same projects as in the portfolio, which is the global optimum for the problem presented in equation (3), obtained with complete information. Also the value of can not be higher than the value of the. holds as well [4] Portfolio Selection Rules Practices for selecting portfolios vary greatly in different organizations [5]. Organizations have developed different selection rules that fit their particular needs. These selection rules may differ in respect to both the number of evaluations that they require per project proposal and in the construction of the final project portfolio that is to be pursued. In following discussion we have to make some assumptions in order to generalize and simplify the process. In practice, a DM has some amount of subjective data about the performance of the project proposals, but the question is, how should she use this information to make a portfolio selection. There are some quite general ways to organize the portfolio selection process. In this assignment we assume that all the information about the performance scores of project proposals are based on expert evaluations. Extensive research about using expert evaluations in decision support [6] has been carried out. There are two main categories of methods for selecting a portfolio: a 1-stage and a multi-stage method. In this assignment we concentrate on the 1-stage and the 2-stage selection processes. If the DM has one evaluation on each project proposal, these evaluations are the best estimates for the real values. In order to have any reason to get more than one evaluation the DM should have some method for calculating and aggregating scores from multiple evaluations in such a way that they are likely to converge to the real values. This means that the DM should have some assumptions about the distribution of the evaluations and also some means of calculating the unbiased estimators for that distribution [7]. 6

7 In this assignment we introduce a new concept in selection rules, namely the concept of time frame. This means that the DM has limited amount of time to make the portfolio selection process. Introducing time to the selection process creates a new risk [8] to the DM; do the expert evaluations arrive in time? The time concept creates new elements that have to be taken into account when choosing a portfolio selection strategy Time Frame In many real life situation the DM has only a certain limited amount of time to make a decision. In our portfolio selection problem time will change the instant decision process to a process, in which the time aspect is greatly involved. The part, which includes most of the time delay in portfolio selection process, is the expert evaluation. There are several parts in the evaluation process that take time. For example, it takes time for the expert to complete his or her evaluation and to deliver/mail the evaluation to the DM. This process can be considered a Poisson's process [9], when referred to the DM's point of view. Based on that assumption, it is possible to expect that there will be a certain number of evaluation statements after a certain period of time. The expert evaluations are a delivery process and therefore can be modeled as a Poisson's process. Figure 1 Illustration of time frame Let us denote new variables to the portfolio selection problem, where is the time the DM needs to complete the portfolio selection, is which is a time needed for single evaluation of a project proposal. (6) If the DM does not have any information about the project proposal at the time of decision, the estimator is set to zero. In the model, it is assumed that all other actions, e.g. delivering information and calculation estimates, in the selection process are instant. 7

8 2.4. Time, Accuracy and Price The three parameters that determine expert evaluations are time, accuracy, and cost. These parameters clearly describe the key elements that affect the selection problem. Time is a crucial factor if the time to deliver the evaluation to the DM is longer than the time the DM has in her disposal to use in the selection process. In such a case that piece of information is rendered useless to the DM, because it would not be available at the time when the portfolio decision is made. Time also affects the expert if he or she does not have enough time to make a good assessment/evaluation of the project proposal. In that case his or her evaluation might be biased and inaccurate, and that misinformation might affect the whole portfolio selection process. It can be assumed that the experts are pursuing evaluations that are as accurate as possible [10]. The cost [14] of the expert evaluation is a main parameter that the DM can use to affect the quality of the expert evaluation. The price affects both the time and the accuracy of the evaluation. The three mentioned parameters form a joint distribution for expert evaluations. This distribution assumption includes an assumption that we can estimate with some sort of distribution to time, accuracy, and price respectively. Time and price can be estimated from real statistics, but the estimation of accuracy is more complex [15]. A possible method to evaluate the expert evaluations is, for example Bayes' method that assesses the expert s accuracy. However, in many real life situations this method can be impractical. In this study, it is assumed that the 3 mentioned parameters form a joint distribution that can be used to estimate the estimators for these 3 model parameters. 3. Simulation Model Performance of the different portfolio selection strategies is a problem that is introduced in previous the chapters. It is hard to test strategies in practice, because the real expected performance of the projects is not known before the decision, and after the decision, only selected projects are observed. Also it is hard to obtain a data to reach statistical result. Results of the selection process can have an impact far into the future. Because of these reasons, we use simulation to examine strategies. The main focus of the simulation is to introduce time relevancy in the portfolio selection problem and try to make it more understandable. The simulation model is built so that it is an illustrative example of real life situation. With this illustrative example we wish to identify, what kind of phenomena DMs could face with the portfolio selection problem, and also introduce possibilities and tools that our simulation model enables. 8

9 3.1. Example Background Story For A Situation Where Portfolio Selection Problem Can Occur An energy company is looking for a set of locations in which to build solar power plants. The company has decided to use expert evaluations to rank the location candidates. The decision regarding locations has to be made before the release of the next annual report, because this new strategic movement for the company has to be announced there. This can be considered as a typical situation in which to use portfolio selection Simulation Parameters Let us simulate a general selection problem, where there are, possible project proposals. The true value of the project proposals is, where. The values are assumed to be continuous real variables in range. An objective thirdparty expert using this same scale of 1 to 5, where the grade five is assigned to the best performing/most suitable, evaluates the project proposals. The evaluations are thought to be continuous and discrete. For simplicity we see no reason to separate the project proposals in terms of cost. As a result we can assign a unit cost to all projects (this might be a strong assumption in real world, but it does not have any significance in the decision process). In the simulation we have possible project proposals. With a budget of units, we choose a subset of 20 projects. The true performance of these projects is modeled with a beta distribution [11] with a mean of 3.3 and a variance of 0.8. The average project performs reasonably well with these parameters. Betadistribution is usually defined in the closed interval. This interval has been scaled to the interval. (7) where and are parameters which define the shape of the distribution. The expected value of and variance are connected to the shape parameters according to the following equations: 9

10 (8) (9) The expert evaluation model is a little bit more complex. The evaluations have to fulfill the following conditions in simulation to be acceptable: 1. The evaluations have to be stochastic. 2. The evaluations have to be based on the true value of the location, i.e.. 3. The evaluations have to belong to closed interval. 4. The evaluations can be continuous or discrete. 5. The evaluation s arrival times have to be. The expert model in simulation is based on the joint distribution of expected time of arrival, accuracy, and cost. This distribution is assumed to be a three dimensional, log-normal distribution. This joint distribution describes the fundamental relationships between the three model parameters. Partial purpose of this simulation is to introduce a method to estimate these three parameters. Lognormal distribution [11] is often used to in economics to model the value delivered by an investment. It assumes only positives values, but on the other hand it does not limit the maximum pay-off. The lognormally distributed accuracy, i.e. error coefficient, has some good qualities as well. In order to satisfy previously presented conditions, the expert evaluations are modeled with a refined multiplicative model. Instead of perturbing the real values of projects, we perturb the ratio of segments that the true values divide into the interval. This ratio is defined by mapping which maps the true values of location to a set of positive real numbers as follows (10) The values are perturbed by multiplying them with a log-normally distributed random variable. In the time frame model comes from the multi log-normal distribution. Total cost is defined as (11) If the j:th location has a true value of, we have following evaluation 10

11 (12) where the is defined in equation (10) and as mentioned above. Estimators for, and are (13) (14) (15) (16) (17) The Expert Evaluations Joint Distributions Parameters Expert evaluations form a joint distribution of three parameters: cost, accuracy, and expected arrival time of the evaluation. In the simulation model, we assume that there are three different kinds of joint distributions; one for low, normal, and high quality expert evaluations respectively. These simulation distributions are generated heuristically for simulation purposes only and it is assumed that these distributions can be determined statically. In the simulation, we create a multi-normal distribution that we use to generate a multi log-normal distribution. The expected arrival time is an open parameter, because it is the one of the main risk factors in the model. Table 1. Joint distribution parameters Quality of Expert Evaluation Low Normal High Expected Price (%) Expected Accuracy Variation of Accuracy and Time It is assumed that it is possible to acquire an expert evaluation with any of the quality and in any expected arrival time mentioned above. The assumption may not be realistic, if the expected arrival 11

12 time is very short. The very short expected arrival time makes it impossible to have as good an evaluation as in a longer time. 4. Results The simulation was done in a scenario were the DM has 10 units of time to finish the portfolio selection. She decides that she will use one third ) of the available time for the 1st stage of the 2- stage evaluation process. This division of time can be changed. The ordered expert evaluations have an expected time of arrival of one third of the possible time available for that particular part in the decision process. The number of expert evaluations that is assigned to the first stage in of a 2-stage model, is either a constant value 2 or it has been varied more realistically. The decision, which kind of strategy is used to determine number of evaluations in certain phase, is one of the main questions when considering optimal portfolio selection strategy. (18) (19) where ceil is a rounding rule to round fractions upwards. When, for example n=[0,5], the total number of evaluations is [2,6], the number of evaluations in the first stage [ ] and the second stage [ ] in a 2-stage model. This kind of strategy emphasizes the importance of the first stage in a 2- stage process. (20) 12

13 Figure 2 Simulation results for 'low' quality evaluations with sufficient time and constant number of evaluation in the first stage. 13

14 Figure 3 Simulation results for 'low' quality evaluations with sufficient time and varying number of evaluation in the first stage. At first, the DM orders low quality evaluations for the 1-stage and for both stages in the 2-stage process. The expected arrival time of evaluations is shorter than the time to make a decision. Figure 4 Number of dismissed optimal project proposals, constant number of evaluations in the first stage in 2-stage process Figure 5 Number of dismissed optimal project proposals, varying number of evaluations in the first stage in 2-stage process ( low quality evaluations) Figures (2) and (3) show that the 1-stage process seems to perform slightly better than the 2-stage process. One reason behind this can be seen in figures (4) and (5) where the average number of rejected optimal project proposals in the first stage in 2-stage process is shown. This causes decrease of performance in 2-stage model and it is more visible when the number of evaluations is varying fig. (3). In the case of constant number of evaluations, the costs of those dismissed evaluations are constantly diminish the value of selection process. Comparing these two processes to a random selection where the average value of the portfolio is percent of the value of the optimal portfolio. These processes are stunningly efficient Figures (2) and (3) part (II) as both of them reach almost 100 percent of the value of the optimal portfolio even though Figures (2) and (3) part (I) show that the number of selected project proposals is not as near as in the projects in the optimal portfolio. Figures (2) and (3) part (IV) show that the value of the investment diminishes when the DM spends more resources in evaluation costs. Let us next examine higher quality expert evaluations; first normal quality. 14

15 Figure 6 Simulation results for 'normal' quality evaluations with sufficient time and constant number of evaluation in the first stage. Figure 7 Simulation results for 'normal' quality evaluations with sufficient time and varying number of evaluation in the first stage. 15

16 Figure 8 Number of dismissed optimal project proposals, constant number of evaluations in the first stage in 2-stage process ( normal quality evaluations ) Figure 9 Number of dismissed optimal project proposals, varying number of evaluations in the first stage in 2-stage process ( normal quality evaluations) Figure (6) part (I) shows that the difference between the 1-stage and the 2-stage process is nonexisting. Similar phenomena can be seen in Figure (7). Comparing the low quality evaluations in Figures (6) and (7) part (I), we can see that with same costs, the percentage of selected optimal projects in the portfolio is lower than with higher quality evaluations, but there is no difference in percentage of value of optimal portfolio in part(ii). Let us next examine high quality expert evaluations. 16

17 Figure 10 Simulation results for 'high' quality evaluations with sufficient time and constant number of evaluation in the first stage. Figure 11 Simulation results for 'high' quality evaluations with sufficient time and varying number of evaluation in the first stage. 17

18 Figure 12 Number of dismissed optimal project proposals, constant number of evaluations in the first stage in 2-stage process ( high quality evaluations ) Figure 13 Number of dismissed optimal project proposals, varying number of evaluations in the first stage in 2-stage process ( high quality evaluations) Similar to the lower quality evaluations, the difference between the 1-stage and the 2-stage processes in Figures (12) and (12) part (I) is small; even smaller in fact than in normal or low quality evaluations. Figures (13) and (11) part (I) show that when the 2-stage process is not missing critical data in the first stage, it performs well. All three different quality evaluations seem to perform quite well, if we compare the percentage of the value of the optimal portfolio. Surprisingly low evaluations have the highest ratio percentage per cost ratio. Let us next examine the situation where time of expected arrival time of the expert evaluations is the same as the time the DM has to make a decision. In the 1-stage process she has 10 time units; 3 time units in the first stage in 2-stage process and 7 time units in the second stage of 2-stage process. Let us see, how the number of ordered evaluations compensates the expected number of missed evaluations. In the simulation it is expected that the DM does not have to pay for missed evaluations. Therefore, the results are affected in a way that encourages the DM to acquire the maximum number of evaluations to minimize the risk of having no evaluation what so ever. 18

19 Figure 14 Simulation results for 'normal' quality evaluations with no time to spare and varying number of evaluation in the first stage in 2-stage process Figure 15 Number of dismissed optimal project proposals, varying number of evaluations in the first stage in 2-stage process ( normal quality evaluations), no spare time. Figure (14) part (I) shows that on average the 1-stage process is performing better than the 2-stage process in situations where the expected arrival time of the expert evaluations is the same as the time to make a decision. The main reasons behind this can be seen in Figure (15) where the average number of optimal projects proposals in the first stage in a 2-stage process is very high as long as long as there is high probability to miss an evaluation of the project proposal. Typically around 4 optimal project proposals are dismissed in the first stage of a 2-stage selection process. This is a fairly high number of all projects, considering that the optimal portfolio includes 20 projects. Surprisingly the value of the selected portfolio is still nearly 95 percent of the value of the optimal portfolio. In the simulation, the project proposals values are generated from the beta-distribution with a mean of 3.3, 19

20 which in turn lead to the fact that quite a few generated project proposals are performing well. It can also be seen in Figure (15) that, when the number of ordered evaluations compensates the expected number of missed evaluations, the 2-stage process begins to increase its performance. We can also see that ordering more evaluations is somewhat pointless if we know that they will arrive too late in any case. Therefore we can conclude that expert evaluations should add value [13] rather than diminish the value. However, when the cost of evaluation increases over a certain limit value, the selection process becomes as good as random selection. Similar characteristics are shown also with low and high quality expert evaluations. Let us next analyze how different expected times of arrival of evaluations affect the outcome. The simulation is done so that it simulates different expected arrival times between 1-15 time units and uses normal quality for evaluations. The DM orders 4 evaluations in both selection processes. In the 2-stage process the 4 evaluations are evenly divided in between stages. Figure 16 Simulation results for different expected evaluation times. Shown as a function of price ('normal' quality evaluations) 20

21 Figure 17 Simulation results for different expected evaluation times. Shown as a function of quantity ('normal' quality evaluations) Figure 18 Simulation results for different expected evaluation times. Shown as a function of relative score ('normal' quality evaluations) 21

22 Figure (17) shows how the simulation results develop as a function of expected arrival time. In the figure, it should be taken into account that for the 1-stage process the time to make a decision is 10 time units and for the 2-stage process the time is divided so that first stage is 3 time units and the second stage 7 time units long. Figure (17) part (I) shows the results for a situation where the first stage in a 2-stage process is kept constant. As the expected arrival time and the number of missed expert evaluations increase both value and the percentage of optimal projects in a portfolio decrease shown in Figure (17) parts (II) and (III). When comparing the performance of 1-stage and 2-stage selection processes, the 1-stage process seems to perform better than the 2-stage process. In Figure (17) parts (II) and (III) the difference is statistically significant when we use the one-way ANOVA with 95 percent confidence. It is a bit surprising that the 1-stage process performs better than the 2-stage process. There is a hidden fact that might affect the outcome; namely in the 1-stage process the number of ordered evaluations is greater and therefore the risk that the combined expert evaluation is biased is smaller than in the 2-staged process. This results from the assumption that the expert evaluations are unbiased. The law of great numbers states that, if we were able to obtain an infinite number of evaluations, the calculated estimate will converge to the true value. Figure (16) parts (I) and (II) show that when considering the value and the percentage of optimal projects in the portfolio, the performance of the 1-stage process is a bit better than the 2-stage process. In Figure (16) part (III) both processes return values that are over the estimate for the values of the project proposals. They are therefore a bit too optimistic. In Figure (16) part (IV) we can see that at a certain point, where the evaluation costs increase over a certain limit, the value added by the expert evaluation becomes less than the cost of acquiring evaluations. Figure (18) part (II) shows that the cost per evaluation in a 1-stage process is almost constant even though in this simulation the DM does not have to pay for delayed evaluations. In Figure (18) part (I) we see that the average cost for a 1-stage process is higher than for a 2-stage, but in both cases the cost are decreases as the number of delayed evaluations increase (this is not true unless the DM does not have to pay for delayed evaluations). Figure (18) part (III) shows the cost of an evaluation process per value of the portfolio. We can see that the 1-stage process performs better than the 2-stage process, but when the expected arrival time of the evaluations lengthens, both processes become more and more like random selection and valueless. We can notice that the 2-stage process varies less than the 1-stage process. The 1-stage process is statistically better than the 2-stage process when using the one-way-anova with 95 percent confidence. 22

23 Figure 19. Simulation results for different expected evaluation times. Shown as a function of price. Figure 20. Simulation results for different expected evaluation times. Shown as a function of relative score. Let us examine the first stage of a 2-stage model. We see some quite predictable phenomena by varying the expected arrival time of expert evaluations,. Figure (22) shows that the performance of the 2-stage process decreases significantly when the first stage suffers from delayed evaluations. In this 23

24 simulation run the 1-stage process parameters are kept constant. The performance is difference clearly seen in Figure (23) where performance is plotted as a function of the expected arrival time. We see that when the expected arrival time of evaluations is closer to the maximum time to make a decision, the performance of the selection process deteriorates. In Figure (23) part (I) we can see that the number of dismissed optimal projects increases as the number of delayed evaluations increases. Figure (23) part (II) shows that the percentage of optimal projects in the selected portfolio also decreases as the number of delayed evaluations increases. The average performance of a random selection is near 70 percent. Thus we can conclude that when the percentage of optimal projects in the selected portfolio falls near 70 percent, the 2-stage process becomes as good as random selection. In Figure (23) part (III) we see that the absolute value of the portfolio does not change a lot but it is statistically weaker than the 1-stage process. Figure 21. Simulation results for different expected evaluation times in the first stage of a 2-stage process.shown as a function of price. 24

25 Figure 22. Simulation results for different expected evaluation times in the first stage of a 2-stage process.shown as a function of quantity. Figure 23. Simulation results for different expected evaluation times in the first stage of a 2-stage process.shown as a function of relative score. 25

26 In Figure (24) part (III) we see that the relative scores for a 2-stage selection process do not perform well. This is caused by the lack of evaluations or biased evaluations in the first stage. Let us next examine which kind of selection strategy would be optimal. Let us first examine a situation where the expected arrival time of expert evaluations is varying. In this case the DM decides to try optimize the cost she knows that 2-stage method is vulnerable in the first stage. She decides to order two normal quality evaluations in the first stage of a 2-stage process, and two low quality evaluations in the second stage. For reference, it can be noted that if she ordered three normal quality evaluations in a 1-stage process, the total costs for evaluations in both processes would be the same. Figure 24 Simulation results for equal costs different qualities in evaluations (Plotted as a function of costs) 26

27 Figure 25 Simulation results for equal costs different qualities in evaluations (Plotted as a function of expected arrival time) Figure (24) shows that the 2-stage process is over-performing compared to the 1-stage process with ANOVA with 95% confidence, and it is on average cheaper as well. It can be also seen in Figure (25), where the simulation results are plotted as a function of expected arrival time of the expert evaluations. The 2-stage process is affected less when the expected arrival time increases Figure (25). In this simulation only the evaluation quality was altered, but to pursue the optimal strategy quality and number of the evaluations, evaluation distribution between first and second stage in 2-stage process can be altered. All this creates a great number of different kinds of possible optimal combinations. Based on the results, it seems that there is no one optimal selection strategy. Instead, it seems that the DM can choose, depending on the situation, a strategy that outperforms all other processes. Also it seems that if we measure the value of the selected portfolio, all methods seem to perform rather well. The one critical observation from the results is that it is better to have some kind of evaluation rather than to have none. Let us now take a closer look at the confidence intervals of a percentage of projects in the optimal portfolio. We want to see how the intervals develop while the number of iterations increases. 27

28 Figure 26. Simulation result confidence intervals. Figure 27. Simulation result confidence magnitudes. 28

29 As we can see in Figure (25) the mean of the percentage of projects in the optimal portfolio has some of variation. However, when we consider the magnitude of the confidence interval Figure (26), we see that it converges quite quickly towards a point estimate. The mean also converges to a point estimate as the number of iteration increases Limitations of Simulation In the simulation there are some limitations that render the model somewhat simplified; namely, we have to determine some parameters, e.g. the joint distribution of accuracy, expected arrival time of expert evaluations, and price before the simulation run. In the simulation runs where time is the interesting variable, we have to determine the number of ordered expert evaluations and the time that is used for the first stage of a 2-stage process beforehand. It is possible to vary these parameters in the simulation model but that will increase the number of free parameters rapidly. The approximation of three different accuracy scenarios is also a bit of restriction, and the number of possible combinations of different cases is unpractical. The quality parameters for expert evaluations are also discrete since there are only three different quality levels. The quality that the DM chooses is used in all evaluations at one stage. This is bit rigid and it might be more realistic, if for example the 1-stage process had some number of low, normal, and high quality evaluations each instead of only one quality. 5. Summary and Conclusions The risk in using expert evaluations in portfolio selection is great when a time frame involved. However, the risk can be controlled. The most important part of this risk is to notice it and take it into account when planning the portfolio selection process. The most important thing is to determine what will be the criterion to which the portfolio selection process is based on. As was seen before, both the 1-stage and the 2-stage process perform superiorly in comparison to random selection. However, in some certain unfavorable situations their performance might decrease near the performance of a random selection, e.g. when the 2-stage process is suffering missed or biased evaluations in the first stage. Thus time creates new limitations to the selection problem; though it can help focus and structure the selection process in more efficient manner as well. In the study the 1-stage method seems to perform well in general, but it suffers a bit from the cost structure and also from the fact that there is no possibility to obtain different quality evaluations for it. The 2-stage process can be seen to be a more delicate method to select a portfolio. The performance of the 2-stage process varies as it can be astonishing or greatly below average. This study also shows that recognizing time dependency is the key to working efficiently in a time dependent environment. However, the study fails to provide any ultimate strategy that would lead to good portfolio selections, but it does introduce some key elements that are present in good decisions. Obviously, the expert evaluation, even if it is perfect, is worthless to the DM if it is not available at the time of decision. This encourages to acquire some, even incomplete, but available evaluations, for 29

30 every selection problem. The 2-stage and other multi-staged processes seem to be vulnerable in the first stage or when the preliminary categorization of the largest number of project proposals is done. This emphasizes the importance of the first stage. It can be concluded that there can be many different, but equally good strategies to perform portfolio selection. It is only a matter of criterions or focuses which kind of process the DM should use to support her decision in a time dependent situation. This study invoked several new questions; the most interesting being, does an optimal, general strategy that would be suitable in most situations exist, and if it does, what it would be. Though the question of existence is not as fascinating in practice. We are more interested in finding a strategy that is near optimal. In this study, simulation was done mostly in certain points of the parameter space, but the generalization to continuous parameters can also be achieved. One question that also arises is that, if we know that we have to make a decision in a certain amount of time, what kind of evaluations would be optimal. In this study, the available evaluations were pre-determined. 30

31 6. References [1] J. Nissinen, The Impact of Evaluation Information in Project Portfolio Selection, Master s thesis, Helsinki University of Technology, (2007) [2] D. Bertsimas, J.N. Tsitsiklis, Introduction to Linear Optimization, Athena Scientific (1997) [3] D. Kleinmuntz, Resource Allocation Decisions. In W. Edwards, R.F. Miles and D. von Winterfeldt, editors,, Advances in Decision Analysis, Chapter 20, Cambridge University Press (2007) [4] J.E. Smith, R.L. Winkler, The Optimizer s Curse: Skepticism and Postdecision Surprise in Decision Analysis, Management Science, 52, No.3, (2006) [5] A.D. Henriksen, A.J. Traynor, A Practical R&D Project Selection Scoring Tool, IEEE Transactions on Engineering Management, 46 No. 2, (1999) [6] R.T. Clemen, Combining Forecasts: A Review and Annotated Bibliography, International Journal of Forecasting, 5, (1989) [7] P.A. Morris, Decision Analysis Expert Use, Decision Analysis, 1 No. 3, (2004) [8] Mohammad Modarres, Risk Analysis is in Engineering: Techniques, Tools and Trends, CRC Press, (2006), ISBN [9] R. Durret, Essentials of Stochastic Processes, Springer, (1999) [10] Decision Analysis, URL: [11] Pertti Laitinen,Todennäköisyys ja sen tilastollinen soveltaminen,otatieto Oy Yliopistokustannus University Press Finland,(2002) [12] S.R. Watson, R.V. Brown, The Valuation of Decision Analysis, Journal of the Royal Statistical Society Series A, 141 No. 1, (1978) [13] Ashish Arora and Andrea Fosfuri, Pricing Diagnostic Information, Management Science, 51 No. 7, (2005) [14] Jeffrey Keisler, Value of Information in Portfolio Decision Analysis, Decision Analysis, 1(3): ,(2004) [15] R. T Clemen and R. C. Kwit, The Value of Decision Analysis at Eastman Kodak Company, ,Interfaces, 31(6):74-92,(2001) 31