USING CONTESTS FOR ENGINEERING SYSTEMS DESIGN: A STUDY OF AUCTIONS AND FIXED- PRIZE TOURNAMENTS

INTERNATIONAL DESIGN CONFERENCE - DESIGN 2016 Dubrovnk - Croata, May 16-19, 2016. USING CONTESTS FOR ENGINEERING SYSTEMS DESIGN: A STUDY OF AUCTIONS AND FIXED- PRIZE TOURNAMENTS A. M. Chaudhar, J. D. Theknen and J. H. Panchal Keywords: crowdsourcng and fundng, game theory, systems engneerng (SE) 1. Introducton: contests n engneerng systems desgn Recently, there has been an ncreasng nterest n usng open nnovaton mechansms, such as crowdsourcng, wthn product development processes [Innocentve.com]. Crowdsourcng nvolves outsourcng of tasks to a large network of unafflated ndvduals outsde the tradtonal organzatonal boundares [Howe 2008]. Employng onlne communtes through the Internet has made t possble to brng n ntellectual captal from outsde as needed [Chesbrough et al. 2006]. Crowdsourcng s typcally carred out n the form of contests where a problem s announced, along wth a prze amount for the best soluton. The use of crowds for nnovatve desgn and problem solvng s not new. For example, tournaments have been carred out snce 1700 s. The Brtsh government, n 1714, offered a monetary prze to come up wth a way to measure a shp s longtude. Sub-contractng by large organzatons such as GE [2009] and NASA [2001] s carred out usng ths approach. It s well known that the soluton qualty s dependent on the ncentves, and the structure of the contest [Terwesch and Xu 2008]. For example, ndvduals may not partcpate f the prze s low, or they may nvest less effort f large numbers of other partcpants are partcpatng, because of the reduced wnnng probablty. Desgnng good crowdsourcng contests necesstates understandng how ncentves affect soluton qualty, how ndvduals decde whether or not to partcpate n the contest, how much effort to nvest, etc. There s currently a lack of understandng on how to organze such crowdsourcng and open nnovaton actvtes for engneerng systems desgn [Panchal 2015]. Whle fundamental research on usng crowdsourcng for desgn s very lmted, there has been sgnfcant effort on modellng research contests n the lterature on research contracts, and procurement of nnovaton. Ths has resulted n a well-developed theory of contests [Corchón 2007]. The theory of contests can provde nsghts for desgnng good crowdsourcng contests. The lterature on desgn contests has not yet been suffcently leveraged wthn the desgn communty. We beleve that ths s due to a combnaton of dfferent factors: () crowdsourcng s a relatvely new phenomenon n desgn, () the lterature on research contests s focused on ssues that have not been at the core of desgn research, and () the nature of research processes s dfferent from the nature of the desgn processes. We have two goals n ths paper. The frst goal s to establsh the relevance of the contest models to engneerng desgn. We dscuss the characterstcs of desgn contests that are smlar to research contests, and the characterstcs that are unque. We show that models of research contests can serve as a startng pont for understandng how to leverage crowds for engneerng desgn. The second goal s to crtcally DESIGN INNOVATION 947

evaluate the assumptons and recommendatons from the models, and ther applcablty n the desgn context. Based on the crtcal analyss, we dentfy open research ssues that need to be addressed for utlzng contest models wthn desgn. The paper s structured as follows. In Secton 2, we provde an overvew of game theoretc models of contests. The specfc focus s on two types of contests: fxed-prze tournaments and auctons. In Secton 3, we dscuss a set of representatve models, and the nsghts provded by these models for desgnng crowdsourcng contests. In Secton 4, we analyse the comparatve use of auctons and contests wthn engneerng desgn. Fnally, the lmtatons of the models and the open research ssues are dscussed. 2. Game-theoretc models of contests and ther relevance to engneerng desgn The use of crowdsourcng contests wthn dfferent parts of the desgn process was recently analysed by Panchal [2015]. The author argues that there are dfferent types of ntatves that can be used n engneerng desgn (e.g., contests, open calls, mcro-tasks, etc.). Desgn contests can be desgned n many dfferent ways, such as () sngle stage vs. multstage tournament, () open entry vs. restrcted entry vs. entry fee, () fxed prze vs. performance based prze vs. auctons, (v) wnner-takes-all vs. multple przes vs. aucton-style tournaments. Addtonally, there are many dfferent crtera wth whch these ntatves can be evaluated. Examples of these crtera nclude the soluton qualty, the number of contrbutors, the amount of effort nvested by the contestants, the qualty of teams formed, the overall cost of runnng the contest, the probablty of gettng a good soluton, and the cost of flterng good solutons. In the rest of the paper, we focus on one of these decsons decdng whether to conduct a fxed prze tournament or an aucton. In fxed-prze tournaments, the wnner's prze s fxed before the start of the tournament. Generally, the prze s fxed by the sponsor to restrct hs/her expendture. The advantages of fxed prze tournaments are that they are smple to mplement and effectve for sponsors wth lmted nformaton about the problem [Fullerton and McAfee 1999]. On the other hand, n contests wth auctons, each bdder submts hs/her best qualty wth a bd of the amount to be pad f the desgn s selected. The sponsor evaluates qualtes and correspondng bds from dfferent contestants, and decdes the wnner. The advantage of auctons over fxed prze tournaments s that auctons reduce the sponsor s burden of determnng the rght prze amount before the contest [Fullerton et al. 2002]. Addtonally, the bddng process provdes an addtonal medum through whch the contestants compete wth each other. The effects of auctons and fxed prze tournaments on the performance metrcs can be analysed usng game-theoretc models of contests. The analyss starts wth dentfyng two types of players, the sponsor and contestants. The sponsor declares the problem to the contestants and notfes them of the type of contest (fxed-prze vs. auctons). Both the sponsor and the contestants are assumed to maxmze ther own expected payoffs. The payoff of each contestant s dependent on the prze amount ( ), the probablty of wnnng the prze ( f ), and the costs ncurred ( ). The probablty of wnnng s dependent on the qualty of ther own submsson ( x ), and the qualty of submssons from other contestants. The cost s dependent on factors such as the nature of the problem and the number of experments carred out. For contestant, gven the prze ( ), the expected payoff s E. After the contestants have submtted ther solutons, the sponsor evaluates the C. f x submssons and chooses the wnnng entry that maxmzes hs/her payoff ( ). 2.1 Auctons vs. fxed prze tournaments The specfc dfferences between the models of auctons and fxed prze tournaments are due to the dfferent payment structures. The prze ( ) for the wnner n a fxed-prze tournament s decded at P the start of the tournament, and s known to all contestants. The wnner, here, s the contestant wth the best qualty. In auctons, contestants prvately submt bds ( p ) along wth ther own best qualty ( x ). Therefore, the wnner n auctons s decded based on the maxmzaton of sponsor s payoff, Es whch s dependent on both qualty and prce. E s 948 DESIGN INNOVATION

In auctons, the sponsor s nterested n better qualty and cheaper prce. The sponsor s payoff functon s generally smplfed by assumng qualty to be the sponsor's monetary valuaton of a soluton. Ths allows drect numercal comparson between qualty and prce whle evaluatng the payoff, E s. Therefore, the payoff takes a smple form, referred to as surplus ( s x p). The sponsor s assumed to decde the wnner after evaluatng each contestant s surplus, x The surplus s, therefore, the p. decson varable for the sponsor and the scorng functon for contestants. Many artcles exst n economcs and management scence lterature that have focused on studyng the detals of auctons and contests [Taylor 1995], [Fullerton et al. 2002], [Che and Gale 2003], [Schottner 2008] n the context of research contests. These models evaluate the best strategy for the partcpants n equlbrum, and predct the expected surplus from the partcpants. The nsghts from the models ndcate that the best desgn of the ntatve depends on the desgn problem and the assocated tasks. Terwesch and Xu [2008] have shown, for example, that desgn problems wth well-defned goals wth no uncertanty n tasks requre the ncluson of sklled contestants n a project. On other hand, problems wth no clear specfcatons, leadng to uncertanty requre large dversty n contestants. Snce these models have been developed for a research context, t s mportant to understand the underlyng assumptons, and the applcablty of these models n the desgn context. Based on our revew of the lterature, we found that these models are based on assumptons about the nature of the process and the problem. Specfcally, the process of research s ether consdered to be sequental or parallel, whle the problem can have dfferent levels of uncertanty. In Sectons 2.2 and 2.3, we dscuss these dfferences and ther mplcatons on usng these models n the engneerng desgn process. 2.2 Process: sequental vs. parallel Some game-theoretc models are based on sngle-perod nnovaton contests [Che and Gale 2003], [Schottner 2008] whle others such as [Taylor 1995], [Fullerton et al. 2002], [Sha et al. 2015] focus on mult-perods contests. In sngle-perod contests, the strategy to come up wth a qualty depends only on the cost of nvestment for that perod. On the other hand, n mult-perod contests, the strategy s also dependent on the outcomes and strateges n the prevous perods. The cost functon x n sngleperod contests s only a functon of the qualty x resultng from a gven perod, whle n mult-perod contests, t s a functon of the number of pror perods (labelled as e ). In mult-perod contests, f a cost per perod s a constant value, C, the total cost s Ce. These assumptons about the research contests relate to the dfferent types of engneerng desgn processes. As Smon [1996] argues, engneerng desgn can be vewed as a search process where desgners search for feasble alternatves, acqure nformaton about them through expermentaton, and select the best alternatve for the desgn problem. These search processes can be broadly categorzed nto parallel search and sequental search [Loch et al. 2001]. The assumpton of ndependence between searches at dfferent steps relate parallel search to sngle-perod contests, whle the dependence of searches on prevous steps relate sequental search to mult-perod contests. In parallel search, the search strategy at each step s ndependent of one another. It s also referred to as samplng, wheren desgners randomze the search to explore the desgn space. Ths approach s generally employed n problems where (a) the uncertanty assocated wth the desgn space s hgh, (b) the cost assocated wth search s low, and (c) the tme requred to complete the search s low. Here, the desgners have more flexblty n testng multple sample ponts. For example, n user nterface desgn, customers changng demands for usablty requre desgners to create multple desgn alternatves cheaply and quckly. In sequental search, the nformaton acqured n the prevous steps s used to gude the search n the subsequent steps. In such processes, decsons about search n subsequent steps are made based upon learnng from prevous steps, stoppng the search at the soluton where the margnal mprovement n qualty s outweghed by the margnal cost to conduct next search [Powell and Ryzhov 2012]. Also, when the cost assocated wth a search and the tme requred are hgh, sequental search s employed n order to reduce the uncertanty quckly. For example, n product development, once the product DESIGN INNOVATION 949

specfcatons are fxed, the desgn proceeds wth the sequence of searches such as concept desgn, testng, detaled desgn, manufacturng and further testng, refnement, etc. Each step depends upon the nformaton ganed n the prevous step. 2.3 Problem: determnstc vs. stochastc The game theoretc models of contests use assumed qualty functons ( x ) or cost functons ( x ) to relate the cost of nvestment to the qualty output. Such relaton can be determnstc or stochastc. In determnstc form, the qualty output s a determnstc functon of number of perods e [Sheremeta et al. 2012], [Sha et al. 2015] or cost () x [Che and Gale 2003], [Schottner 2008]. Ths case assumes a complete control over the qualty output. In stochastc form, on other hand, the qualty output s a cumulatve dstrbuton of the qualty, F [Taylor 1995], [Fullerton et al. 2002]. Ths case, therefore, assumes uncertanty n the qualty output. Such dstncton of qualty control relates to the nature of the desgn problem. In desgn scenaros, f a desgner puts effort for searches, he/she can arrve at the soluton determnstcally or wth uncertanty. For example, consder a towng tank experment to dentfy the hydrodynamc performance of an ocean vessel as a functon of the Reynolds number. The desgner performs a seres of experments to fnd the resstance of the vessel for varous towng speeds or Reynolds number. The qualty or smoothness of ths resstance varaton depends on the number of experments conducted. Each experment s assocated wth a pre-determned cost or effort level. Such class of experments s labelled as determnstc. In another experment where the desgner s tryng to attan a new varety of steel that meets the requrement of the vessel. The desgner has lmted pror nformaton on the type of materal propertes (or qualty ) that may be attaned by dfferent searches. Here, a search may be usng dfferent carbon content or annealng temperature. Ths class of experments can be referred to as stochastc. Here, the output from nvestng a certan level of effort s not a determnstc qualty, rather a probablstc dstrbuton. Wth ths analyss of relevance between game-theoretc models and engneerng desgn, we dscuss the detals of some models n the followng secton. 3. Specfc models, nsghts, and ther applcaton wthn engneerng desgn We start wth a smple classfcaton of the models (Fgure 1) wth the two categores of desgn processes sequental and parallel, and the types of contests as the other dmenson fxed prze and auctons. 3.1 Sequental search fxed prze Ths category ncludes the problems whose desgn process s vewed as sequental search, and are solved usng fxed-prze tournament. Relevant models n ths category are Taylor [1995] and Sha et al. [2015]. These models assume that partcpants undertake research n perods. Research at each perod s assumed to have a constant cost, C. Taylor [1995] analyses tournaments wth a fxed maxmum perod T. The model assumes at least two contestants ( N 2 ) who have to pay an entry fee ( E ) to partcpate. At each perod, contestants derve the qualty probablstcally accordng the cumulatve dstrbuton functon F x. All the contestants have same dstrbuton F x,.e., they are symmetrc. P, EN,, Tand C are assumed to be common knowledge. However, contestant s qualty of a soluton s prvate between the sponsor and the contestant, and s unobservable by other contestants. 950 DESIGN INNOVATION

Sequental Search Taylor 1995 Sha et al. 2015 Fullerton et al. 2002 Parallel Search Che and Gale 2003 Shottner 2008 Che and Gale 2003 Shottner 2008 Fxed-Prze Aucton Fgure 1. Categorzaton of models Under these assumptons, Taylor determnes the best strateges for the contestants, ncludng, whether to enter the game and when to stop search. Taylor clams that each contestant decdes the next strategy based on the number of contestants and value of ts current best qualty, x. For a general case of T 2, the decson about whether to take an addtonal perod, or not, s based on the net expected mprovement n payoff by an addtonal perod. The expected mprovement for contestant s b, where s defned on, s the cumulatve dstrbuton P [ ( y) ] df y C F [0, b ] ( y) x of the best qualty offered by any of the 's rvals. The contestant stops at the qualty ( z ) when the net expected mprovement of payoff s zero. Ths s referred to as Z-stop strategy,. ( z) 0 Sha et al. [2015] assume that the qualty acheved at specfc perod s a functon of the number of prevous perods. Two choces consdered for such functon are lnear ( x ) and exponental ( e x ). Ths model assumes two symmetrc contestants wth the same functonal form for the exp( e) qualty. In equlbrum, both contestants choose the optmal number of perods n order to maxmze both contestants expected payoffs. Ths s done by applyng frst-order optmalty condton on contestants payoff functons ( E C ) wth respect to the correspondng prevous number of perods. In summary, Taylor provdes the optmal qualty (z-stop) the sponsor can expect from a partcular contestant n fxed-prze tournament n terms of the z-stop strategy. Sha et al. [2015], wth qualty as a functon of number of perods, propose the optmal number of perods contestants should take to maxmze ther payoffs. The optmal qualty n equlbrum can then be nferred from a qualty-perod relaton from ths model. 3.2 Sequental search - auctons In ths type of scenaro, bdders perform research n steps, and at the end of pre-defned number of perods they submt bd prces n addton to the best soluton. The relevant model n ths secton s by Fullerton et al. [2002]. Fullerton et al. extend Taylor s model for the case of auctons. The assumptons and equlbrum characterzed n ths model are same as n Taylor s model. The expected qualty outcome z of contestant from an aucton s characterzed by the z-stop strategy, b, where s the equlbrum bddng prce ( z) [ p( y) ( y) p( z) ( z)] df y C 0 p () y z functon of qualty y. Ths functon s transformed from standard frst-prce, ndependent-prvate values aucton n aucton theory. If (; x z represents the cumulatve dstrbuton of contestant 's best, T) qualty at the end of perod T, and f all N contestants are assumed to be symmetrc n ther qualty x N 1 dstrbuton, the equlbrum bd functon s, [ ( y)] dy 0 p ( ) [McAfee and McMllan 1987]. x N 1 [ ] DESIGN INNOVATION 951

Fullerton et al. show that when the equlbrum bd s monotoncally ncreasng n qualty, the expected cost of the prze n aucton s strctly less than the cost of the prze requred to elct the same effort n fxed-prze tournament. 3.3 Parallel search fxed-prze tournament The models of parallel search wth fxed prze nclude Che and Gale [2003] and Schottner [2008]. These models consder a sngle-perod nnovaton wth complete control over qualty. The models assume that the expected qualty ncreases wth contestant s nvestment. For nstance, f a desgner nvests multple tmes to come up wth a best qualty, the qualty output at each draw wll be ndependent of the results of prevous draws, and only depend on how much nvestment s made for that draw. The model by Che and Gale [2003] assumes a determnstc relaton between the nvestment cost and the qualty x. The qualty s unobservable to other contestants. The cost functon, however, s x assumed to be common knowledge among all players. The model characterzes a mxed-equlbrum to predct the contestants behavour for two partcpants. Gven that the contestants are symmetrc and have knowledge about the other contestant s cost functon, both receve zero expected payoff ( E ) at C 0 equlbrum. Ther qualty dstrbutons at equlbrum are, therefore, F. The sponsor can P expect the qualty outcome x to be an expectaton of qualty wth respect to dstrbutons F and exp 1 F. In order to evaluate the optmal fxed-prze for the tournament, the expected payoff of the 2 Popt sponsor E s s maxmzed by applyng the frst-order optmalty condton, E, where s a functon of, and represents the s max ( Popt) xexp P F xexp x P F exp probablty of wnnng for contestant, whch depends upon the qualty dstrbuton of the other contestant ( ). The expected surplus from the fxed-prze tournament s, therefore, s. exp xexp Popt Schottner [2008] models qualty as a combnaton of a determnstc component (referred to as nvestment strategy) and a random varaton. The nvestment cost of comng up wth a gven qualty s a determnstc functon of nvestment strategy. The nvestment cost, n contrast to Che and Gale [2003], s assumed to be prvate nformaton. But the output qualty of each contestant ( x ) s consdered to be common knowledge among all the contestants. The equlbrum n ths model for a fxed-prze tournament s characterzed for two contestants based on the dfference between ther qualty outputs. Schottner [2008] argues that the large uncertanty n ths dfference, due to a large varaton n a random component of the qualty, can be economcally dsadvantageous for the sponsor. Snce the qualty x s common knowledge, the large dfference n qualtes may elct hgh bd prces n aucton. Therefore, under these assumptons, fxed-prze tournament s a better choce for the sponsor. In summary, for parallel search process n engneerng desgn, Che and Gale [2003] defne the optmal qualty output the sponsor can expect, and the optmal fxed prze for the tournament. In addton, Schottner [2008] suggests that the sponsor should prefer fxed-prze tournament n desgn problems wth hgh uncertanty. 3.4 Parallel search - auctons The models such as Che and Gale [2003] and Schottner [2008] are also applcable n parallel search process wth auctons. These models analyse auctons to answer the questons such as: what s an optmal bddng strategy for contestants? how much payoff do bdders acheve? what s the expected surplus from an aucton? and do auctons perform better than fxed-prze tournaments? The assumptons n Che and Gale [2003] model for auctons are the same as the assumptons dscussed n Secton 4.3 for fxed-prze tournaments. The model characterzes the mxed-equlbrum strateges for two symmetrc contestants n an aucton who receve zero expected payoffs n equlbrum. Ths model assumes a complete control over choosng a qualty. In equlbrum, the contestants strategy s to choose the optmal qualty x that mnmzes the other contestant s surplus dstrbuton G ( s) at equlbrum, opt p 952 DESIGN INNOVATION

G. The expected surplus, thus, s an expectaton wth respect to and, () s mn(x )[ ] G1( s1) G2( s2) opt x s.e., s. The model results n the optmal bddng prce functon from exp E s s sand s s contestant at equlbrum as: p. ' n It can be nferred from Che and Gale s model that f s of form, where s an nteger greater x / k n than 1, and k s a postve constant, an aucton always provdes better surplus than a fxed-prze tournament., here, satsfes the condtons of the model by Che and Gale [2003],.e., t s a convex, ncreasng functon, s zero at zero qualty, and crosses a threshold qualty x such that. thd ( xthd ) xthd Schottner [2008], as dscussed n Secton 4.3, suggests that a fxed-prze tournament domnates a frstprce aucton f qualty s common knowledge and uncertanty n qualty submssons s hgh. 4. Analyss of models for the context of engneerng desgn 4.1 Examples of applcatons n engneerng desgn Innovaton contest models from economcs and management scence lterature are based on assumptons about contest settngs, problem specfcaton, behavour of partcpants n equlbrum, and exchange of nformaton. The models dscussed n Secton 4 are summarzed n Table 1. These models provde desgn-specfc nsghts n terms of the expected soluton output from the contest and the cost of conductng a contest. Consder an example of the FANG challenge by DARPA [2010], whch nvolved an onlne contest to desgn an amphbous transport vehcle. The fxed prze $1 mllon was awarded to the wnnng desgn. Although partcpants, non-tradtonal defence engneers, n such contests requre certan level of expertse, the uncertanty s hgh because of ther unfamlarty wth the sponsor s (DARPA) mltary requrement of a more-ntegrated vehcle. Ths deaton-based project, thus, can be categorsed as fxed-prze parallel search contest. The qualty of submssons was prvate. Assumng the valdty of the model by Che and Gale [2003] for the FANG challenge, optmal qualty from ths contest would be x exp gven n Secton 3.3. Also, the model suggests that havng an aucton nstead of fxedprze n FANG challenge would have been proftable. If, however, the desgns were observable to others before bddng n aucton, Schottner s model suggests that fxed-prze would be better than auctons. Topcoder.com, also employs fxed-prze tournaments for challenges for software development. The challenges, submtted by IT companes (sponsor) are open to partcpants who are requred to possess suffcent programmng sklls. Wth well-defned problem statements and sklls requrements, the contests fall under the category of fxed-prze sequental search contests. Also, the qualty of soluton s almost a determnstc functon of how much effort the coder puts n the development. As the qualty s a prvate varable, Taylor s model apples n ths case wth qualty dstrbuton F x varyng n a small range to satsfy near determnstc trend. Taylor model predcts the expected qualty (z-stop) from these contests accordng z-stop strategy. Fullerton et al. [2002] model suggests that havng an aucton nstead of fxed-prze s proftable for IT companes. If, however, the qualty was observable by others, the optmal qualty output from the contest and the optmal effort requred from the partcpants s gven by Sha et al. [2015]. Model Taylor [1995] Desgn Process Sequental Table 1. Model assumptons and results Payment Qualty Cost Equlbrum Results Structure Fxedprze Stochastc; pror dstrbuton assumed; Prvate Constant value per try; Bdder stops at specfc try gven by Z- stop strategy Expected qualty (z-stop) from fxedprze contest DESIGN INNOVATION 953

Fullerton et al. [2002] Che and Gale [2003] Schottner [2008] Sha et al. [2015] Sequental Parallel Parallel Sequental Fxedprze and Aucton Fxed- Prze and Aucton Fxed- Prze and Aucton Fxed- Prze Stochastc; pror dstrbuton assumed; Prvate Determnstc; Full control over qualty usng nvestment; Prvate Determnstc and Stocha-stc parts; Determnstc component dependent upon nvestment; Determnstc; Qualty s a functon of effort (number of tres); Constant value per try; Determnstc functon of qualty; Determnstc functon of qualty + constant cost for a contest; Prvate Constant value per try; Bdder stops at specfc try gven by Z- stop strategy Symmetrc bdders offer the same surplus (two bdders) For small qualty dfference, bdders (two) offer the same surplus; For large dfference, hgher qualty gves hgher surplus Bdders (two) get maxmum payoff; frst order optmalty condton on payoff wrt effort Aucton costs lesser to acheve the same expected z-stop Expected surplus hgher n aucton than fxed-prze For sgnfcant dfference between qualtes, fxedprze preferred than aucton Number of tres by bdders, and correspondng qualty output n equlbrum 4.2 Research gaps The economc behavour of contest models s largely unclear n real-world scenaros. The dffculty n understandng ths relatonshp, frst, arses due the n-approprateness of ther assumptons about qualty and cost n desgn scenaros. Second, behavour of partcpants n actual contests may not be ratonal, as assumed whle analysng the equlbrum behavour. Specfc research gaps nclude: 1. Determnstc relaton between the soluton qualty and nvestment n research, assumed n many contest models, s generally unknown n engneerng desgn. 2. The degree of nfluence of desgn expertse on the contest outcomes s not extensvely modelled n the lterature. The models such as [Terwesch and Xu 2008] propose to tackle expertse. However, these models decouple the cost from expertse. Based on expertse and experence of the desgner, and accessblty to technologcal nfrastructure, the cost ncurred durng product development wll be dfferent for two dfferent contestants. 3. The search process, n most desgn problems, s a combnaton of parallel and sequental unlke the rgd assumptons n the contest models. In the ntal desgn stages, uncertanty nvolved s hgh; hence, exploraton and parallel search strategy domnates. In the later stages, wth a reasonable understandng of the desgn space, the sequental strategy domnates. Models such as [Loch et al. 2001], [Erat and Krshnan 2013] analyse ths combnaton, but ts effects n contest desgn remans unexplored. 4. The effect of desgn space on the soluton qualty s not accounted for n the contest models. The queston of where to look for a soluton s largely unanswered. Rather, these models focus on whether to look for a soluton. 5. Closng comments Crowdsourcng contests have been employed by many organzatons (e.g., Innocentve.com, DARPA 2010, Challenges.gov, and Local Motors) to solve engneerng problems. In ths paper, the focus s on 954 DESIGN INNOVATION

two mechansms frequently used n engneerng contests, fxed-prze tournaments and auctons. There s a lack of knowledge n the desgn communty on how the two contests perform n terms of the qualty outcome, cost of prze, partcpaton, etc. Most contests are conducted as fxed-prze to restrct the expenses. Ths requres deeper knowledge of the feld on the part of the soluton seeker to decde the prze amount. Also, dfferent economc models [Fullerton et al. 2002], [Che and Gale 2003] suggest that auctons reduce the expenses of conductng a contest. In ths paper, we categorze engneerng problems based on ther characterstcs such as desgn process and uncertanty n achevng a soluton to transform them nto noted contest models. Further, we arrve at gudelnes for organsng tournaments to solve these problems. The gudelnes are broadly categorzed n three steps: 1. Match the problem to one of the models. The assumptons lsted n Table 1 facltate the matchng of the problem wth a sutable model. 2. Estmate the expected qualty of the output and the expected cost of przes usng nsghts from the model. Once the contest model s dentfed for a problem, ts results provde nsghts nto questons such as whch payment structure to use, what qualty or surplus to expect from the contest, what tme restrants to mpose on the partcpants, etc. 3. Choose the optmal contest (fxed-prze tournament or aucton) for the gven problem. Two examples dscussed n Secton 4.1 elaborate ths step on choosng an approprate contest. Apart from qualty, cost and payment, many other aspects of contests such as entry fees, number of partcpants, and multple-przes need to be analysed from a desgn pont of vew to evaluate ther applcablty n engneerng desgn. In addton, other mechansms such as patent races, matchng between the sponsor and desgners etc. that nfluence nnovatons need to be studed from desgn perspectve to nclude them n manstream nnovaton actvtes. Acknowledgments The authors gratefully acknowledge the fnancal support from the US Natonal Scence Foundaton (NSF) through grants 1265622 and 1400050. References Che, Y., Gale, I., "Optmal Desgn of Research Contests", The Amercan Economc Revew, Vol.93, No.3, 2003, pp. 646-670. Chesbrough, H., Vanhaverbeke, W., West, J., "Open Innovaton: Researchng a New Paradgm", Oxford Unversty Press New York, 2006. Corchón, L. C., "The theory of contests: a survey", Revew of Economc Desgn, Vol.11, No.2, 2007, pp. 69 100. DARPA, "Adaptve Vehcle Make", Avalable at <http://cps-vo.org/group/avm>, 2010. Erat, S., Krshnan, V, "Managng Delegated Search over Desgn Spaces", Management Scence, Vol.58, No.3, 2013, pp. 606-623. Fullerton, R., Lncster, B., McKee, M., Slate, S., "Usng auctons to reward tournament wnners: theory and expermental nvestgaton", RAND Journal of Economcs, Vol.33, No.1, 2002, pp. 62-84. Fullerton, R., McAfee R., "Auctonng Entry nto Tournaments", Journal of Poltcal Economy, Vol.105, No.5, 1999, pp.573-605. GE, "Supplers Home", Avalable at <https://www.gesuppler.com>, 2009. Howe, J., "Crowdsourcng: Why the Power of the Crowd Is Drvng the Future of Busness", Crown Busness, 2008. Loch, C., Terwesch, C., Thomke, S., "Parallel and Sequental Testng of Desgn Alternatves", Management Scence, Vol.47, No.5, 2001, pp. 663-678. McAfee, R., McMllan, J., "Auctons and Bddng", Journal of Economc Lterature, Vol.25, 1987, pp. 699-738. NASA, "Award Fee Contractng Gude", Avalable at <http://www.hq.nasa.gov/offce/procurement/regs/afgudee.html>, 2001. Panchal, J., "Usng Crowds n Engneerng Desgn Towards a Holstc Framework", In Proceedngs of the 20th Internatonal Conference on Engneerng Desgn (ICED15), Poltecnco D Mlano, Italy, 2015. Powell, W. B., Ryzhov, I. O., "Optmal Learnng", John Wley & Sons Inc., 2012. DESIGN INNOVATION 955

Schottner, A., "Fxed-prze tournaments versus frst-prce auctons n nnovaton contests", Economc Theory, Vol.35, 2008, pp.57-71. Sha, Z., Kannan, K., Panchal, J., "Behavoral Expermentaton and Engneerng Systems Desgn", ASME Journal of Mechancal Desgn, Vol.137, No.5, 2015, pp. 051405. Sheremeta, R., Maters, W., Cason, T., "Wnner-Take-All and Proportonal-Prze Contests: Theory and Expermental Results", ESI Workng Paper, 2012. Smon, H., "The Scences of the Artfcal", The MIT Press, 1996. Taylor, C., "Dggng for Golden Carrots: An analyss of Research Tournaments", The Amercan Economc Revew, Vol.85, No.4, 1995, pp. 872-890. Terwesch, C., Xu, Y., "Innovaton Contests, Open Innovaton, and Multagent Problem Solvng", Management Scence, Vol.54, No.9, 2008, pp. 1529-1543. Jtesh H Panchal, Assocate Professor Purdue Unversty, School of Mechancal Engneerng 585 Purdue Mall, 47906 West Lafayette, Unted States Emal: panchal@purdue.edu 956 DESIGN INNOVATION