Bidding for Complex Projects: Evidence from Italian Government s Acquisitions of IT Services

Transcription

1 Bidding for Complex Projects: Evidence from Italian Government s Acquisitions of IT Services Gian Luigi Albano, Federico Dini, and Roberto Zampino Consip S.p.A., Italian Public Procurement Agency, Via Isonzo 19/e, Roma, Italy gianluigi.albano@tesoro.it, federico.dini@tesoro.it, roberto.zampino@tesoro.it Abstract. Public buyers are often mandated by law to adopt competitive procedures to ensure transparency and promote full competition. Recent economic literature, however, suggests that open competition can perform poorly in allocating complex projects. In exploring the determinants of bidding behavior in tenders for complex IT services, we find results that appear consistent with theory. Our analysis shows that price and quality are linked in a puzzling way: high quality is associated with low prices. We also find that quality proposals are mainly explained by suppliers experience. Results suggest that scoring rules at the basis of the tendering process might fail to appropriately incorporate buyers complex price/quality preferences. Keywords: procurement, tenders, scoring rules, IT contracts, complex projects. 1 Introduction Public buyers often use competitive bidding to select contractors. Competitive bidding was already popular among private procurers, and today is a central principle in public procurement regulations worldwide, in particular in Europe and in the U.S.. Competition in fact allows buyers to pursue cost minimization and to ensure transparency. However, when procurement involves a complex project, economic theory suggests that competition may not be the best allocation mechanism ([9] Goldberg 1977, [12] Manelli and Vincent 1995). Empirical evidence on construction procurement, for instance, highlights the potential limits of sealed-bidding: these stem from unexploited communication between buyer and suppliers ([3] Bajari, McMillan and Tadelis, 2008) and to the difficulties in capturing post-contract adaptation costs ([5] Bajari and Tadelis, 2001). One critical point for the buyer is to precisely describe (many) quality dimensions that are often unverifiable and that can sometimes be only partially known in the bidding stage (e.g., functionalities of a new software to be developed). More precisely, the issue is how to design a scoring auction, i.e., how to appropriately incorporate the buyer s price/quality preference in the tender design. Scoring auctions 1 i.e., tenders in which both 1 Henceforth we will use competitive bidding, competition, tenders and auctions as synonymous. Scoring auctions, which refer to the most economically advantageous criterion of the European public procurement legislation, are used to define competitive process in which both price and quality matter. M.A. Wimmer et al. (Eds.): EGOV 2009, LNCS 5693, pp , Springer-Verlag Berlin Heidelberg 2009

2 354 G.L. Albano, F. Dini, and R. Zampino price and quality matter, are quite common in the practice of public procurement. They are particularly appropriate for commercial/standardized items ([6] Che, 1993), such as laptop and printers, that is when quality can be easily measured and well captured by a suitable scoring rule. Scoring auctions instead may not perform well in complex projects as they force the buyer to give a precise shape to the scoring rule to account for complicated, often unknown, price/quality preferences. In Italy, the Ministry of Economy and Finance (MEF) selects the contractors of IT services through scoring auctions. Since 1997 Consip operates in the behalf of the MEF, being in charge to organize tenders and award contracts. IT services contracts are often general purpose, i.e., they include a large variety of activities, such as maintenance of systems and applications, developments of new applications, IT consultancy. Quality proposals consist in providing effective and flexible teams of professionals and technological solutions to best fit the various needs of the MEF. The selected contractor is called to shape the several elements of the proposed projects to best meet the buyer s needs. In this paper we explore the suppliers behavior in bidding for complex IT projects. After presenting some descriptive statistics, we address the issue of potential consequences of using competitive procedures when theory advises against doing that. More precisely, our research questions are: i) can observed bids tell us something about how well the awarding mechanism captures the buyer s price/quality trade-offs ii) what is (if any) the relationship between price and quality? To answer these questions we exploit a unique dataset of contracts for professional IT services that Consip awarded on behalf of the MEF. In particular, we use the complete set of 20 contracts awarded by Consip in the period in the sector of IT development and consultancy. We find no evidence of a tension between price and quality in submitted price/technical bids: data exhibit a puzzling negative correlation between quality and price, such that higher quality is associated to lower prices. Although not conclusive, the results put at least some doubts on the suitability of scoring auctions for capturing complex price/quality trade-offs. Regression analysis also shows that incumbency is an important determinant of ex-ante submitted quality, suggesting that past experience plays a key role in explaining bidding behavior. The rest of the paper is organized as follows. Section 2 surveys some recent empirical literature. Section 3 describes the institutional context and the contract awarding process. Section 4 overviews the dataset and presents some basic statistics. Section 5 illustrates the results of the empirical analysis. Section 6 draws the concluding remarks. 2 Related Literature This paper is related to the empirical literature on bidding in procurement auctions. There are several papers exploring important issues in (public) procurement. For instance, [4] Bajari, Houghton and Tadelis (2007) and [5] Bajari and Tadelis (2001) estimate the importance of renegotiation and adaptation costs in the procurement of highways paving works in the U.S.. When projects are complex and the contractual design is incomplete, softening competition e.g., negotiating with a restricted pool of suppliers, may be appropriate for selecting the most efficient supplier. This view finds support also in [3] Bajari, McMillan and Tadelis (2008), who compare auctions with negotiations by examining a comprehensive data set of private sector building contracts in the U.S.

3 Bidding for Complex Projects: Evidence from Italian Government s Acquisitions 355 There is, however, lack of empirical investigations on how scoring auctions (scoring rules) and other key factors such as incumbency affect bidding behavior. First attempts to explore such issues are in [11] Lundberg (2005), although in a completely different setting of legal regulation of tender procedures, and in [13] Zhong (2007) who explores some key issues in online procurement auctions for manufacturing goods from a large buyer in the high-tech industry. He characterizes the suppliers bidding behavior to examine the effect of incumbency on bidding. His most interesting finding is the biased towards the incumbent suppliers, but in a setting where (in contrast to our case) the buyer is not committed to a specific scoring rule. 3 The Institutional Context Consip was created in 1997 with the mission to provide the MEF with ICT solutions, technologies and services. One important task of Consip is to manage ICT acquisitions to evolve and maintain the whole IT infrastructures supporting the MEF activities. Consip is today mandated to: define needs/solutions, organize tenders, appoint evaluating committees, evaluate suppliers proposals, award and manage contracts, and finally, monitor suppliers performances. In compliance with the EU Directive 2004/18 all these contracts are awarded through open competitive procedures, usually with the most economically advantageous tender (MEAT). Quality is crucial for IT services contracts. It often accounts for more than 50% of the total weight. Evaluation of proposals is largely based on discretional components e.g., adequacy of IT solutions and organizational structures to fit the MEF s needs. The typical contract requires that contractor sets up a team of professionals (the chief of the project, senior consultants, junior consultants, function analysts, programmers, etc.), IT equipments and technological solutions to achieve both high quality standards and sufficient flexibility to manage heterogeneous activities. Tenders are evaluated on the basis of following main aspects: i) organization (e.g., how professionals are organized and deployed to best perform tasks); ii) solutions (e.g., software, methodologies and tests for development activities); iii) quality (e.g., quality plans, type-documents released such as users guides). 3.1 Tenders Evaluation: The Role of Committees Tenders are evaluated by ad-hoc committees. The committee judges whether and how the organizational setting and proposed solutions are suitable to perform the various activities provided for the contract. At the end of the evaluation process, the committee assigns the technical scores for each supplier. These scores are disclosed in a public session with suppliers. In the same session the committee opens the sealed envelopes containing price bids and publicly announces the submitted prices. One important point is the composition of the committees: this is regulated by the law. Until 2006 the legislation provided for committees to be mixed, i.e., composed of both purchasing administration s employees (insiders) and external professionals, such as full professor or recognized experts (outsiders). Since 2007 committees consist of all insiders. In our analysis we will account for such change to capture the potential differences in quality evaluation under the two regimes.

4 356 G.L. Albano, F. Dini, and R. Zampino 4 Data 4.1 Bids and Scores Our analysis is based on a unique set of 20 contracts that Consip awarded in the period Analyzed contracts total value amounts to 428,7 millions, 4,6% of total Italian expenditure on IT services in 2006 (private and public sectors amount to 9,3 billions). The simple ranking of contracts by raffled off technical scores 2 shows that quality is very important. Note that more than 60% of contracts are skewed on the technical side. In the majority of contracts quality weighs at least 60%. Contracts in which quality is at least 50% are 85% of total contracts. Symmetrically, the frequency distribution of financial scores shows that 60% of lots has been faced with scores until 40 points, or 85% under 50 financial scores. Table 1. Frequency distribution of available Technical Score α < N % Table 2. Frequency distribution of available Financial Score β >50 N % Table 3 and 4 present the frequency distribution of observed relative scores effectively achieved by competitors. The relative score is equal to Actual _ Score. The Max _ Score central technical score ranges (51-60 and 61-70) represents the 50% of technical proposals, whereas 62% of technical proposals obtained scores over 60. Overall average technical score is 66.17, median is 65 and standard deviation is Statistics also show that there is a limited number of bids (9 out of 132) that are in the highest score range (91-100). Things are quite different if we analyze the frequency distribution of relative financial scores. The first two higher ranks (81-90 and ) together account for the 51% of proposals, while the 75% are over 50. Furthermore, both the mean and the standard deviation (72.60 and respectively) are greater with respect to the technical scores. Price bids exhibit much more dispersion than quality bids, even if bidders seem to achieve higher ranks of financial scores more easily rather than analogous levels of technical scores. 2 Henceforth, we will use score(s) and point(s) interchangeably. This holds also for lot(s) and contract(s).

5 Bidding for Complex Projects: Evidence from Italian Government s Acquisitions 357 Table 3. Frequency distribution of relative Technical Score rank <= N % Summary Statistics Mean Median St. Deviation Table 4. Frequency distribution of relative Financial Score rank <= N % Summary Statistics Mean Median St. Deviation Table 5. Score matrix Winners Technical Score Best score Not best Winners Financial Best score 5 2 Score Not best 11 2 Interestingly, suppliers do not win submitting outstanding financial proposals. The score matrix shows that the winner obtains the highest technical score in 16 cases out of 20, whereas only in 7 cases she gets the highest financial score. This suggests that suppliers mainly win contracts by promising relatively more (ex-ante) quality rather than submitting low price, so much so they win 11 times thanks to the best technical score, without achieving the best financial ones. 4.2 Scoring Rules Contracts for IT services often contain many aspects of quality. Buyers usually deal with such a multidimensional problem by awarding contracts with MEAT criterion. MEATs are concretely performed through scoring rules which transform the price (and/or qualitative aspects) into a score. Thus the contract is awarded to the supplier with the total highest score. As a preliminary analysis, fig. 1 shows how rebates of winners increase on average when the scoring rule is linear with respect to other rules. A scoring rule is linear if the score increases proportionally as the price declines. This type of scoring rule belongs to the family that we may call independent scoring rules: in this case the bidder s score depends on her bid only. Interdependent scoring

6 358 G.L. Albano, F. Dini, and R. Zampino 70,00 60,00 59,97 Rebate % 50,00 40,00 30,00 20,00 10,00 0,00 Average Average 35,03 31,80 40,22 18,02 Average Lowest Lowest Lowest 8,44 8,59 41,66 43,23 35,95 Lowest 49,30 50,01 Scoring rule 40,79 Fig. 1. Rebates of winning bidders and scoring rules 46,12 42,47 43,86 19,63 Average Average Concave 34,60 24,86 21,75 Concave Concave rules, instead, are such that the score of any bidder also depends on some (or all) other bids (e.g., the lowest bid, the highest bid, the average bid, etc.). 3 In our dataset the latter type of rule leads to higher submitted prices on average. Such a difference may stem from the unpredictability of the score for interdependent scoring rules. (and thus predictable) scoring rules, on the contrary, allow suppliers to compute easily the score for any possible price bid. This, in turn, enables them to choose the optimal price/quality bidding strategy. 4 Interdependent scoring rules tend to yield significant lower rebates on average about 27% with respect to 46% than independent scoring rules. In particular, lowest and/or average price-based scoring rules induce suppliers to submit bids as close as possible to what they expect the best or the average price will be. The more precise this estimate the more chances the supplier has in achieving an 3 See [3] Che (1993) and [2] Asker and Cantillon (2008) for theoretical implications and properties of scoring auctions. See also [7] Dimitri, Dini, Pacini and Valletti (2006) for an indepth analysis on the design of scoring rules. The linear scoring rule is a very simple way to transform price bids into a score. This rule is described by [ ] : ( Re serve Pr ice Pr icebid ) Pr icescore = nn, where the price threshold is a percentage of the reserve ( Re serve Pr ice Pr icethresho ld ) price that the procurer may want to introduce in order to stimulate competition on price and nn is the available max financial score. 4 Two things is worth stressing. First, the procurement environment we are considering refers to the IPV (Independent Private Value) model. In this type of (auction) model, project s costs depend on the supplier s intrinsic efficiency (e.g., organizational efficiency). In CV (Common Value) models, instead, costs are influenced by others costs estimates. Our assumption of IPV relies on the idea that the potential heterogeneity in information on the procurement environment affects more the ability of suppliers to write a comprehensive technical proposal than the costs of the projects itself. Second, CV model de facto makes costs/evaluation of the contract interdependent. Such an interdependency should not be confused with that of the scoring rule, which refers to the interaction of suppliers bids when applying a certain type of scoring rule.

7 Bidding for Complex Projects: Evidence from Italian Government s Acquisitions 359 high score. The uncertainty, however, may trigger a precautionary behavior on the price side. Both effects may induce suppliers to bid higher prices and thus to increase purchasing costs for the buyer. The effects of interdependent scoring rules has not been studied systematically by the theoretical literature. First indications from [1] Albano et al. (2007) suggest that interdependent scoring rules might facilitate some form of coordination among bidders. Lower dispersion found in submitted bids when scoring rules are interdependent might not conflict with the authors findings. 5 Empirical Analysis In this section we use the dataset to explore the main factors explaining suppliers bidding behavior. Our estimates are based on a reduced form OLS regression approach, which allows us to focus more on the directions rather than magnitude of the effects. Our reduced form model is the following linear model: y i = α + x i,k β k + ε i. (1) and k = 1, K indexes all our explanatory variables x, while i = 1, N indexes the observation units. A cross-section estimate is carried out on 132 observations submitted for winning 20 contracts (summary of statistics are in table 6). The list of variables includes the following: Number of bids. The number of actual bids proxies the level of the ex-ante expected competition in the tender. In mature markets, as that of IT services, suppliers are likely to know each other, and thus to infer potential participants to a tender. Therefore, bids informs about what suppliers know on the level of competition in the tender. In general, this variable can be an important determinant explaining bidding behavior. Standard auction theory (independent private value) suggests that, all else being equal, prices (competition) tends to increase with the number of actual bids (in procurement, the price decreases with participation). Scoring rules. This is a binary variable, 1 for independent and 0 for interdependent scoring rules, respectively. Our conjecture is that independent scoring rules should stimulate competition on the economic side more than interdependent, since suppliers can calculate ex-ante the change of the score associated to a variation of the price bid. Interdependent scoring, instead, complicates bidding and the conjectures suppliers make about other competitors bidding behavior. In average scoring rules, for instance, suppliers are induced to guess the average price in order to bid as close as possible to it. This tends to concentrate prices around the average guessed level rather than the lowest possible level. Experience. This is measured by the number of previously won contracts for any bid i at any given time t, by each bidder. We expect more experienced suppliers to better know the procurement environment and thus, ceteris paribus, to offer proposals that better fit the various needs of the buyer. Expert suppliers are expected to be better informed about the real MEF s needs and how to put this knowledge into more

8 360 G.L. Albano, F. Dini, and R. Zampino comprehensive technical offers. This should yield higher technical scores with respect to less (or non-)experienced suppliers. 5 Committees. This is a binary variable, equal to 1 for all-insiders committees and 0 for mixed committees (insiders + outsiders). This is a control variable capturing possible technical score variability due to different evaluation approaches under the two types of committee described above. Bids and scores. We use technical scores to proxy the ex-ante quality offered by suppliers. Rebates, financial scores and the price/reserve price ratios are alternative measures of economic effort. Table 6. Summary statistics of variables Variable Mean St. Dev. Min Max Obs Bids per contract Experience Tech. Score (relative) Financ. Score (relative) Rebates Reserve Price 21,5 Mln 1.88e+07 0,491 Mln 70,3 Mln Testing for Price/Quality Trade-Off In this section we investigate whether and how quality is explained by price, controlling for the key elements of the tender design (namely the type of scoring rule and the number of bidders). We measure quality with the suppliers actual technical score. Another research question is to identify possible incumbent effects measured by the suppliers past experience. The equation we estimate is the following: Tech_Score i = const + β 1 Financial_Score i [Rebate i ; Bid_Price/Res_Price i ] + (2) β 2 t Winning it + β 3 N_Bids i + β 4 Scoring_Rule i + β 5 Committee_dummy i + ε i. We performed 5 regressions 6 with alternative measures of the price bid: financial score, rebate and relative price (price bid/reserve price). All regressions suggest that price does explain quality. Coefficients are all statistically significant. However, a 5 Contractor s superior information about the procurement environment does not necessarily imply that the correct theoretical model to analyze data is common value. Talks with practitioners and contract managers, suggests that superior information does not affect significantly the suppliers evaluation of the contract, rather their ability to write suitable and wellcrafted projects/proposals. Therefore, superior information affects the technical score achievable by suppliers via best fitting proposals. 6 Tests indicate that the estimated model is not affected by multi-collinearity for independent variables. F-test indicate that all variables should be included in the regression. Goodness of estimation: despite parsimony, the model is able to explain up 30%-35% of total variance. Further testing rejects the hypothesis of non-normality in estimated residuals, therefore supporting the choice of a linear model for our data. These considerations hold also for the price regressions.

9 Bidding for Complex Projects: Evidence from Italian Government s Acquisitions 361 puzzling positive link is predicted between quality and price: higher quality is associated to lower prices and vice versa. The sign in this relation seems to reject the classical paradigm of a price-quality trade-off in submitted bids. Nevertheless, this could be not so surprising because of some arguments already mentioned above. One first explanation is the difficulty for the buyer to incorporate complex price/quality trades-off in the tender design. Since quality is the ability of professionals and how they are deployed to best perform tasks, it turns out quite hard for the buyer to give precise economic evaluation to quality of such dimensions, and thus shape correctly the scoring rule. Price and quality may thus end up exhibiting a perverse relationship. The results might also be driven by the presence of non-contractible quality considerations. Since many quality dimensions are hard/costly to monitor ex-post, suppliers may anticipate this in the bidding stage and offer low prices for the promise of outstanding quality (yielding high technical scores), which will be a lower ex-post effective quality. A similar effect is studied in theory by [10] Kim (1998). Noncontractible quality implies transaction costs for contract enforcement and difficulties to ensure that the project is of the desired high quality. In this framework the author points out that if the buyer commits himself to a firm fixed price contract, the contractor may provide low quality in order to cut down on production costs. A third explanation is related to discretional evaluation of bids on committees side. This acts as shock. For a given quality a supplier may receive either a lower or higher technical score depending on the subjective evaluation of the committees, therefore altering the true relationship between price and quality. Despite we have introduced a committees dummy for different types of committees, we have not been able to isolate the potential shock effect. 5.2 The Role of Other Variables Estimates suggest that independent scoring (linear and concave) 7 reduces technical score (the sign of coefficients is always negative as reported in table 7). Independent rules allow each supplier to determine his financial score unloosed from his competitors behavior. This provide him with a clear incentive to improve the price offer. It is worth noting, on the contrary, that interdependent rules (lowest bid and average scoring) introduce uncertainty also on the price side. Scores become unpredictable because of the simultaneous presence of both discretional evaluation of technical side and interdependent price scoring on the economic side. In this context, incentives for the suppliers to shift effort from quality towards price improvements are expected to be weaker since they cannot know the rate of return in terms of financial score. 8 7 Concave scoring is such that the score increases less than proportionally as price declines. A standard concave scoring can be as follows: S i = [1 (P i /Pb) α ] PE. Where S i is the score obtained by bidder i, P i is the price submitted by bidder i, Pb is the reserve price, α measures the slope of the curve and PE is the weight of price in the tender. Concave scoring clearly discourages bidders to bid aggressively, as soon as the incremental score is made negligible (depending on α) for marginal reductions of P i. 8 With independent scoring rules such a shift can indeed pay: rather than offering X additional consultants at a cost of say , to get an uncertain incremental technical score, the supplier can easily compute the (certain!) incremental score associated to a price reduction of the same amount.

10 362 G.L. Albano, F. Dini, and R. Zampino Table 7. Technical score regressions OLS. Tech_Score i I. II. III. IV. V. Financial_Score i 0.156*** (3.24) Rebate i *** (3.72) 0.364*** (3.47) 0.244** (2.44) Bid_Price/Res_Price i t Winning it N_Bids i Scoring_Rule i Committee_dummy Constant term 3.07*** (6.55) -1.41*** (-2.86) -5.12** (-2.00) *** (-3.76) *** (16.27) 3.414*** (7.15) ** (-1.97) *** (-3.49) (-1.03) *** (12.62) 3.55*** (7.44) *** (-4.14) (-1.09) *** (13.34) 3.01*** (6.23) *** (-2.91) (-0.65) *** (12.60) *** (-3.72) 3.414*** (7.15) ** (-1.97) *** (-3.49) (-1.03) *** (12.08) Adj. R F-test N. Obs t-statistic shown in parentheses; significant levels at *0.10, **0.05, ***0.01. The variable t Winning it summarizes the number of past contracts awarded to each bidding supplier. Experience/learning is what the supplier have learnt in executing the contract. Learning can be important in complex procurement like the ones we are considering. The incumbent learns how to make a custom tailored proposals and thus to exploit this superior information in subsequent tenders. Winning one additional contract is associated to supplier s improvement of the relative technical score by roughly points, about 6% of relative technical score on average. The number of submitted bids has a negative impact on the technical score. A first possible explanation is that more participation shifts the players efforts towards price-competition rather than technical-competition. Again, the expectation that quality improvements may not be appropriately rewarded (or will do less than price improvements) may induce suppliers to shift effort from quality to price when expecting higher participation. Scoring rule and expected participation appear to interact, and to operate in the same direction. Despite statistical significance is achieved only when using the financial score as covariate, the composition of committees seems to affect technical scores in the conjectured direction. Internal committees are associated to a lower average technical score. Insiders tend to reward quality proposals less than outsiders. One explanation is that risk-averse external members, who typically know less about the procurement context, may over-reward technical proposals to avoid appeals from suppliers.

11 Bidding for Complex Projects: Evidence from Italian Government s Acquisitions Conclusions In this paper we explored the determinants of suppliers bidding behavior, using a unique dataset of contracts for IT services that Consip awarded on behalf of the Italian Ministry of Economy and Finance. Although evidence is based on a small number of observations, the analysis may provide some indications. In a framework of competitive tendering to allocate complex IT projects, we find no tension between price and quality in observed bids. Price and quality appear related in a puzzling way: higher quality is associated to lower prices. This results may be stem from the difficulty for the buyer to adequately describe and incorporate her price/quality preferences into the tender design (and more precisely in the scoring rule). Another finding is that past experience appears to be among the most important determinants of quality proposals. Superior information on the procurement environment can significantly increase the suppliers chances to achieve high technical scores and thus to win the contract. References 1. Albano, G.L., Bianchi, M., Spagnolo, G.: Bid Average Methods in Procurement. In: Piga, G., Thai, K. (eds.) The Economics of Public Procurement. Palgrave (2007) 2. Asker, J., Cantillon, E.: Properties of Scoring Auctions. Forthcoming, Rand Journal of Economics (2008) 3. Bajari, P., McMillan, R., Tadelis, S.: Auctions vs. Negotiation in Procurement: An Empirical Analysis. Journal of Law, Economics and Organization (forthcoming, 2008) 4. Bajari, P., Houghton, S., Tadelis, S.: Bidding for Incomplete Contracts: An Empirical Analysis of Adaptation Costs. NBER Working Paper No (2007) 5. Bajari, P., Tadelis, S.: Incentives Versus Transaction Costs: A Theory of Procurement Contracts. Rand Journal of Economics 32(3), (2001) 6. Che, Y.-K.: Design Competition through Multidimensional Auctions. Rand Journal of Economics 28, (1993) 7. Dimitri, N., Dini, F., Pacini, R., Valletti, T.: Scoring Rules. In: Dimitri, N., Piga, G., Spagnolo, G. (eds.) Handbook of Procurement. Cambridge University Press, Cambridge (2006) 8. Georgia Institute of Technology, zhong_fang_200712_phd.pdf 9. Goldberg, V.P.: Competitive Bidding and the Production of Precontract Information. Bell Journal of Economics 8, (1977) 10. Kim, H.I.: A Model of Selective Tendering: Does Bidding Competition Deter Opportunism by Contractors? Quarterly Review of Economics and Finance 3(4), (1998) 11. Lundberg, S.: Auction Formats and Award Rules in Swedish Procurement Auctions. CERUM Working Paper No. 79 (2005) 12. Manelli, A.M., Vincent, D.R.: Optimal Procurement Mechanisms. Econometrica 63(3) (1995) 13. Zhong, F.V.: Empirical Analyses of Online Procurement auctions: Business Value, Bidding Behavior, Learning and Incumbent Effect. Ph. D. Dissertation, College of Management. Georgia Institute of Technology (2007)