Implicit Learning for Explicit Discount Targeting in Online Social Networks

Transcription

1 Implct Learnng for Explct Dscount argetng n Onlne Socal Networks Srnvas Shakkotta Dept. of ECE exas A&M Unversty Emal: sshakkot@tamu.edu Le Yng Dept. of ECE Iowa State Unversty Emal:leyng@astate.edu Sankalp Sah Dept. of ECE exas A&M Unversty Emal: sankalpsah@neo.tamu.edu Abstract Onlne Socal networks are ncreasngly beng seen as a means of obtanng awareness of user preferences. Such awareness could be used to target goods and servces at them. We consder a general user model, wheren users could buy dfferent numbers of goods at a marked and at a dscounted prce. Our frst objectve s to learn whch users would be nterested n a partcular good. Second, we would lke to know how much to dscount these users such that the entre demand s realzed, but not so much that profts are decreased. We develop algorthms for multhop forwardng of such dscount coupons over an onlne socal network, n whch users forward coupons to each other n return for a reward. Couplng ths dea wth the mplct learnng assocated wth backpressure routng orgnally developed for multhop wreless networks, we show how to realze optmal revenue. Usng smulatons, we llustrate ts superor performance as compared to random coupon forwardng on dfferent socal network topologes. We then propose a smpler heurstc algorthm and usng smulatons, show that ts performance approaches that of backpressure routng. I. INRODUCION he past few years have seen the rapd and global emergence of onlne socal networks as a medum for communty nteracton [1], [2]. her success can be gauged by ther meteorc adopton by a large populace, and the contnued success of the medum requres a sound economc foundaton for sustanable growth. he medum of choce to extract commercal value out of onlne socal networks s the advertsng and sale of goods and servces by usng the structure and nature of socal nteractons. Snce an ndvdual user s preferences can be dentfed by hs or her response to exogenous stmulaton such as advertsements, an approach that s often used s to try to learn about user preferences through exploraton. Such feedback obtaned though the user responses could be used to offer ncentves to purchase certan goods and servces. Consder goods and servces that are consumed perodcally say on a weekly or monthly bass such as move tckets, car washes, ftness club vsts and so on. Here, we could have a hgh dsplayed prce that some consumers would be wllng to pay. In order to extract maxmum revenue, other consumers need to be subsdzed to some extent by usng dscounts such as rebate coupons. In other words, dscount coupons are used to create multple ters of prces for the same good or servce. wo questons mmedately arse, whch users should be gven coupons?, and how many coupons should they be gven? Further, the questons have to be answered n a system n whch user preferences change over tme. Both questons are hard snce the seller of the good s unlkely to be aware of the preferences of users, or possbly even of ther exstence. Even f the seller s aware of a user s nterest, he must not gve too many or too few dscounts too many would reduce profts and too few would mean that the entre demand would not be realzed. here are two classcal methods of offerng such ncentves. he frst s to flood communtes of users n the hope that some of them would use the coupons. Here, the dea s to pre-dentfy communtes that are not lkely to buy the good wthout dscounts. If dentfcaton s ncorrect, ether the users would not use the coupons, or the wrong set of users would be dscounted. he second s to rely on self-dentfcaton of nterested ndvduals. Here, the store gets the users to sgn up for coupons, and then judcously sends them coupons. Such a scheme would work only on users who do dentfy themselves to the store, and mght not realze the entre demand. Both the above solutons gnore the fact that users could belong to an onlne socal network, and hence could obtan coupons by nteractng wth hs or her frends. hus, users could forward coupons from one to the next n a multhop fashon across the onlne socal network. If a user s nterested n good that the coupon represents, he or she could use t. Otherwse the user could forward t onwards. Allowng for coupon forwardng mples that the two questons rased have to be modfed slghtly: gven that a user has a coupon that he does not want, whch frend should he forward t to and why?, and what rate should coupons be njected nto the system? Hence, we need to desgn a sgnalng scheme that ncentvzes users to somehow learn the preferences of users n such a way that the profts of the store are maxmzed. An example of such a system n practce s mgnger [3] that acts as a mult-hop advertsng and dscount dstrbuton system usng SMS messages, wth rewards beng pad n a pyramdal fashon. he motvaton for mult-hop coupon dstrbuton s that snce user preferences change wth tme, and new products are contnuously ntroduced, t s mpossble for any store to be aware of all ts potental customers. Hence a system must learn user preferences, whch then change after a whle. In ths paper, we develop mplct dstrbuted learnng schemes based on deas of backpressure [4] that has been

2 used as a throughput optmal scheme for packet routng n multhop wreless networks [5], [6], [7], [8]. We assume that the capacty for consumpton of a good by user j can be dvded naturally nto two parts one at at the marked hgh prce ˆx h j, and one at the dscounted low prce. We assume that these values are fxed for some duraton ˆx l j of tme, and so can be learned. he number of coupons gven to the user must be carefully regulated; f t s larger than ˆx l j, the store loses profts due to excessve dscountng, whle f t s less than ˆx l j, the entre demand s not realzed. We combne deas of self-dentfcaton by users and drected floodng through backpressure to acheve an optmal soluton that realzes the entre demand by njectng the rght number of coupons, and maxmzes proft by ensurng that the users receve coupons at the optmal rate. We then use optmzaton decomposton technques n Secton IV to yeld three dstnct problems, each of whch can be solved usng local nformaton. Such a decomposton also naturally suggests an archtecture wth a smple controller assocated wth each problem, whch we develop n Secton V. Our system conssts of three enttes: a store at whch goods may be purchased, users connected by an onlne socal network and a coupon source or sources. he behavor of these enttes s as follows: Each store sells goods at a marked prce p, whch t dscounts to a prce q upon presentaton of a coupon. he store assgns a goodness value to each user j that makes a purchase form t. hs value determnes the probablty wth whch neghbors of j are rewarded for forwardng a coupon to j 1. However, all the other users non-neghbors of j that are nvolved n the forwardng path are guaranteed a reward. hs artfce enables localty of nformaton, as we show later. In other words, although the dscount carred by each coupon s dentcal, the reward for forwardng coupons to each user j s not. he store uses a smple up-down controller to determne the reward probablty for forwardng, based on the number of goods purchased by user j. All users n the system mantan a count of the number of coupons of each knd that ther neghbors possess va communcaton over the socal network. Users also mantan a count of the number of unrewarded coupons assocated wth ther neghbors by pollng the relevant store. We refer to the sum of these two as the effectve coupons. Coupons can be transferred among users n a multhop fashon, and users are ncentvzed to forward coupons n the drecton of lowest effectve pressure,.e., to a neghbor who has the smallest number of effectve coupons. hs controller s smlar n nature to a backpressure controller. Fnally, the coupon source generates coupons of dfferent knds correspondng to dfferent goods, and sends them 1 hroughout ths paper, we use the word reward to denote remuneraton for forwardng coupons, and the word dscount to denote remuneraton for redeemng coupons when purchasng goods at a store. to users that have dentfed themselves as nterested n recevng partcular knds of coupons. he source chooses to send coupons usng a threshold controller, whch generates new coupons when the effectve pressure s less than a certan threshold. We prove that the system usng ths backpressure-based coupon dstrbuton evolves wth tme to attan the maxmum profts by ensurng that each potental consumer obtans exactly the rght number of coupons. he system s dstrbuted and each user only requres nformaton assocated wth hs or her neghbors. hus, t succeeds n achevng lght-weght learnng framework, n whch explorng for user capacty and explotng exstng capacty go hand-n-hand. We then consder a smpler heurstc algorthm n Secton VI, that s based on the delay n obtanng rewards. hs delay-based algorthm does not requre nformaton exchange between users. At any tme, users smply forward coupons to that neghbor for whom the delay experenced between forwardng a coupon and obtanng a reward for that coupon s the smallest. hs algorthm nherently captures the dea of backpressure, although t s at a coarse level. Our fnal scheme s even smpler, and conssts of random coupon dstrbuton. Here, each user randomly forwards coupons to ts neghbors n the hope of fndng correct paths. hs system does not learn user preferences. We use ths algorthm to test the effcacy of our other algorthms. We smulate the dstrbuton schemes Secton VII on dfferent topologes to compare ther performance. We show that the backpressure-based scheme acheves near-optmal revenue, whle the delay-based scheme performs acceptably well. Further, both schemes sgnfcantly outperform the randomzed scheme, thus makng a strong case for backpressure based targeted coupon delvery n onlne socal networks. II. RELAED WORK A typcal learnng problem s that of the bandt-problem [9], [10], [11], whch has ts hstorcal orgn n a con-operated gamblng machne that pays off accordng to the matchng of symbols on wheels spun by a handle, also called a one-armed bandt. he mult-armed bandt s the stuaton confronted by a gambler who has a choce between n one-armed bandts, and attempts to learn by experence whch one to pull. Our approach to learnng s qute dfferent, and follows a more mplct method derved from communcaton networks. he backpressure algorthm s a jont routng/schedulng for communcaton networks where the routng/schedulng decsons are dynamcally made wthout requrng the nformaton of the network topology and traffc arrvals [4]. he algorthm has been proved to stablze any traffc that can be stablzed by any other rotung/schedulng algorthm for dfferent types of communcaton networks [12], [13], [14], [6], [5], [15], [16]. he backpressure algorthm learns traffc and network nformaton mplctly from queue-lengths. Based on the smlar dea, we develop a coupon delvery scheme, where the users learn the coupon demands from coupon queues and dynamcally dstrbute coupons to ther neghbors based on

3 the effectve pressure. However, coupon delvery problem has fundamental dfferences from routng/schedulng for communcaton networks, whch nclude: user behavor follows user belefs, so a proper ncentve scheme s needed to motvate users to dstrbute coupons n the rght way; users are selfsh and wll use all the coupons they get, so a good coupon delvery scheme should avod sendng too many coupons to a user. hs mples that a user that s nterested partcular type of coupon cannot serve as a relay for that type of coupon. hs s fundamentally dfferent from communcaton networks where every node can serve as a relay. Fg. 1. III. SYSEM MODEL A coupon dstrbuton system Network model: We consder an onlne socal network structure as shown n Fgure 1. Denote by N the set of nodes and L the set of lnks. here are three dfferent nodes coupon dstrbutor, users, and store n the network. he lnks represent socal connectons. A lnk from a coupon source to a user represents the dea that the user has regstered wth the source to receve ts coupon perodcally. A lnk from a user to a product means that the user buys that product perodcally. he lnks between users are assumed to be bdrectonal, and represent frendshp between the connected users. In ths paper, we assume that the coupon sources and the store are managed by the same entty. We use s to denote the store and d to denote the coupon dstrbutor. We consder a synchronzed slotted-tme system. We defne µ j to be the coupon transmsson capacty of node j, whch s the maxmum number of coupons user j can send out n one tme-step. We also mpose the constrant that a user can buy a dscounted product only f a coupon s presented. wo-capacty model for user demands: We assume that users naturally have a maxmum number of goods that they would buy at the marked prce, ˆx h j, and number of goods that users would buy at the dscounted prce, ˆx l j. Note that ether of these quanttes could be zero. We further defne b j = ˆx h j +ˆx l j. hese values can be thought of as the capactes assocated wth a user. We consder two dfferent tme scales n ths paper. he small tme scale t s the one n whch purchases are made and coupons are delvered. he large tme-scale, consstng of small tme slots, s the buyng nterval after whch the customers start afresh. Snce the users have the ncentve to buy a product wth a low prce, we assume that the ˆx h j, assocated wth the hgh prce goods could be used to buy low prced goods as well. Specfcally, durng each large tme scale, f the user were gven no more than b j coupons, he would use them all and buy b j goods. Note that f a user were gven more than ˆx l j coupons, the store would not extract the maxmum extractable revenue. If he were gven less than ˆx l j coupons, he would not buy enough dscounted goods, whch reduces the proft of the store as well. A store that s unaware of these two capactes needs to probe customers n order to fnd ther true potental, and nether supply too few or too many coupons. In what follows, we present a dstrbuted soluton that automatcally explores for and attans the capacty of users, thus achevng the proft maxmzng soluton. IV. PROFI MAXIMIZAION Consder the proft made by the store. We say a coupon s vald f t s eventually used to purchase a product. We denote by ym,n the average number of vald type- coupons sent from user m to user n n a tme slot. he proft the store extracts from user j s { } q yj,s + p mn ˆx h j,b j yj,s. hus, the maxmum proft the store can extract s defned by the followng optmzaton problem: OP 1: s.t. max S S j B { } q yj,s + p mn ˆx h j,b j yj,s 1 ym,j µ m, m N 2 j:m,j L,j s ym,j = yn,m m N 3 j:m,j L n:n,m L y m,j =0f m B and j s 4 where 2 s the capacty constrant, whch ndcates node m cannot send more than µ m coupons n a tme-slot, 3 s the flow-conservaton constrant for the coupons, and 4 ndcates that user j wll not forward type- coupons to hs/her neghbors f he/she uses coupon. o extract the maxmum revenue, we need to dstrbute coupons to the users. here are two dffcultes n dstrbutng the rght number of coupons to the users: h 1 he optmal ˆx j, ˆxl j s unknown at the store, and needs to be dentfed. 2 Snce all users nterested n a product may not be regstered to drectly receve coupons, they mght need to receve such coupons va the socal network. he store cannot drectly control the number of coupons sent to such users. o tackle these two dffcultes, we develop a two tme-scale coupon dstrbuton scheme n the next secton. V. COUPON DISRIBUION In ths secton, we develop an mplct dstrbuted learnng scheme based the dea of backpressure routng/schedulng

4 [17]. Our algorthm conssts of two control loops that operate at the small tme scale and the large tme scale. he purpose of the control loops s as follows: 1 Choce of Coupon Forwardng Reward Rate: At the large tme scale, each store must adapt the target rate γj for the next buyng nterval usng the nformaton gathered about the customers preferences over the past ntervals. In our algorthm, γj s an estmate of ˆxl j. As dscussed n Secton III, f γj s set too low, customer j may not purchase all the goods that he potentally could, and f γj s too hgh, customer j may be beng dscounted excessvely and the store s not extractng the maxmum extractable revenue. 2 Coupon Routng at arget Rate: At the small tme scale a store arbtrarly assgns to a rate of coupon delvery γj to each product and each customer j that purchases goods from t. he purpose of ths control loop s to ensure that the customer would ndeed receve dscount coupons at ths target rate. Mathematcally, we wll guarantee that the coupon dstrbuton algorthm solves the followng optmzaton problem: OP 2: s.t. S max S q j B y j,s 5 ym,j µ m, m N 6 j:m,j L,j s ym,j = yn,m m N 7 j:m,j L n:n,m L y j,s γ j j, 8 y m,j =0f m B and j s 9 o show the correctness of the proposed algorthm, we frst need the followng straghtforward lemma. Lemma 1: Gven that γj =ˆxl j for all and j, OP 1 s equvalent to OP 2. Proof: he proof s presented n Appendx A. Next, we develop a dstrbuted coupon routng algorthm that solves OP 2. A. Small tme scale control: backpressure coupon routng We frst ntroduce the coupon management scheme whch conssts of three parts: each user mantans a per-product queue, and montors the lengths of the queues; store rewards the neghbors that forwarded type coupons used by each customer j at a rate γj, and montors the number of unrewarded coupons at each customer 2 ; coupon dstrbutor montors the number of coupons she has not yet sent out, and generates addtonal coupons based on ths value. A1: Coupon Management: 1 Per-product queues are mantaned at each user, and the number of type coupons user j has at a fner tme-step 2 Recall that these are coupons that have been redeemed for a dscount by j, but the neghbors of j who forwarded these coupons have not been rewarded. t s denoted by Q j [t]. hus, the dynamc of Q j [t] s as follows: If j B, then = otherwse Q j[t + 1] Q j[t]+ Q j[t + 1] = Q j[t]+ where y j,s m:m,j L m:m,j L [t] = mn Q j[t]+ b j y m,j [t] n:j,n L y m,j [t] y j,s [t], m:m,j L + t yj,s [τ], τ=0 y m,j [t], yj,n [t].e., user j wll use up all avalable coupons unless she has already bought enough b j products. 2 Store mantans a queue for unrewarded coupons correspondng to each of product and ts customers j. We may thnk of these as vrtual coupons that are used to mantan a pressure on j s neghbors. Note that t s only the neghbors of j that are not rewarded for forwardng these coupons, the rest of the users nvolved n forwardng coupons would be guaranteed a reward and, of course, j has already redeemed these coupons for a dscount. Denote by Q j [t] the number of such unrewarded coupons correspondng to customer j. We have + Q j[t + 1] = Q j [t]+yj,s [t] γ j, where γj s the coupon forwardng reward rate for neghbors of customer j. 3 Coupon dstrbutor d mantans a queue for the coupons that have not been sent out. he length of the queue s denoted by Q d [t] for each product. We have Q d[t + 1] = Q d[t] + Θ [t] + yd,j [t], j:d,j L where Θ [t] s the number of new type coupons generated by coupon dstrbutor at tme t. 4 We also assume that when user j receves a type coupon such that j B, user j wll nsert her dentty and the coupon queue length Q j [t] n the coupon before sendng the coupon to her neghbor. hs nformaton allows the store to reconstruct path and reward the coupon relays based on Q j [t]. In our system the store need to reward coupon dstrbutors n order to motvate users to forward coupons to ther frends. he effcency of a coupon dstrbuton scheme s determned + ;

5 by: the ncentve scheme that the store use, and the users decsons under the ncentve scheme. Next, we propose a coupon rewardng scheme, under whch a ratonal user wll dstrbute the coupons accordng to a backpressure polcy. he optmalty of the coupon dstrbuton scheme wll be proved n heorem 2. A2: Reward Scheme for Coupon Forwardng: Store rewards the users nvolved n forwardng each used type coupon wth a total of α dollars. Consder a specfc coupon assocated wth product, and assume R s the path consstng of the sequence of transmssons used to dstrbute the coupon over whch the coupon was transferred. hen node m gets a reward Q m Q n:m,n R + α l R Q sl Q rl +, 10 where l s a lnk on path R, sl s the sender, and rl s the recever. Note that ths queue length nformaton s nserted by the users before they forward the coupons to ther neghbors. Furthermore, note that the amount of reward user m obtans s proportonal to the queue dfference. he dea s to motvate user m to send her coupon to a neghbor who has the least number of coupons and hence s most lkely the one who needs the coupon. Under ths scheme, the user has the motvaton to follow the backpressure-lke coupon dstrbuton scheme. A3: User Behavor: 1 Frst, f node j s nterested n buyng product, then user j uses all avalable type coupons up to her purchasng lmt b j. hus, at fner tme-step t, user j purchases yj,s [t] products wth coupons such that yj,s [t] = mn Q j[t]+ ym,j [t], m:m,j L + t b j yj,s [τ], τ=0 2 If user j s not a customer buyng product, then user j needs to dstrbute type coupons to her neghbors. We assume that at the begnnng of fner tme-step t, user j requests Q m[t] f user m s her neghbor, and also polls the store to obtan Q m[t]. Snce the amount of coupon forwardng reward s determned by the queue dfference as descrbed n 10, user j selects a coupon type and neghbor m such that,m arg max Q j[t]+ Q j[t] Q m[t] Q m[t], j,m L and transfers mn { µ j,q j [t]} of type coupons to node m. Note that Q m [t] s the number of coupons used by user m but not been rewarded yet, so Q m[t] = 0 f user m s not a customer buyng product. A store mantans ths unrewarded coupon queue to prevent a customer recevng too many coupons. When user j uses too many coupons, the unrewarded coupon queue becomes large. After a neghbor of user j fnds a large Q j [t], the neghbor realzes that user j has receved too many coupons and the store mght not reward hm for forwardng coupons to user j. hen the neghbor wll stop forwardng more coupons to user j. A4: Coupon Generaton Scheme: Each coupon dstrbutor needs to decde the number of coupons to nject nto the network. We assume that coupon dstrbutor generates µ d type- coupons when Q d [t] Q q, and zero type- coupon otherwse. Here, Q s a constant threshold value. In other words, Θ [t] =µ d f Q d [t] Q q, and Θ [t] =0otherwse. In the followng theorem, we analyze the performance of the backpressure coupon routng, and prove that heorem 2: Assume that γj b j for all and j. Under the coupon management, coupon rewardng and generatng scheme, and user behavor defned above, we have lm lm t=1 Θ [t] = ˇy j,s, 11 Q j B and lm lm Q t=1 y j,s [t] =ˇy j,s, 12 where ˇy s the optmal soluton of OP 2. Proof: he proof s presented n Appendx B. Note that although the theorem s an asymptotc result, the algorthm tself works for any value of. A fnte value of may result n a sub-optmal soluton. In our smulatons, we choose = 300 and the fnal coupon allocaton s very close to the optmal. B. Large tme scale control: Coupon rate selecton We assume that the algorthm for coupon delvery at the small tme scale converges quckly to the target rate, and now consder how to choose ths target rate. Recall that our means of mplementng coupon delvery at rate γj s to reward the neghbors of a customer j for forwardng coupons to j at rate γj. In ths secton all dynamcs take place at the large tme. hus, we have the sequence of target rates γj [0],,γ j [k 1],γj [k],γ j [k + 1],, and the large tme scale algorthm needs to guarantee that lm k ˆγ j [k] = ˆxl j. Denote by zj h [k] and zl j [k] the number of product that user j buys from store n the nterval [k, k + 1] at the marked prce and the dscounted prce, respectvely. Let the total number of goods purchased n the nterval [k, k + 1] be denoted zj [k] = zh j [k] +zl j [k]. Further, denote the dfference n purchases made by user j over ntervals [k, k+1] and [k 1,k] by zj [k] =z j [k] z j [k 1] correspondng to a dfference n the coupon delvery rate γj [k] =γ j [k] γ j [k 1]. We frst ntutvely understand the four possbltes assocated wth γj [k], x j [k] assumng that γ j [k] s small :

6 γj [k] < 0 and z j [k] =0:hs mples that γ j [k] ˆx l j and the user s recevng more coupons than he can use. We need to ensure γj [k + 1] <γ j [k]. γj [k] < 0 and x j [k] < 0:hs mples that γ j [k] < ˆx l j and the user s not recevng enough coupons to realze the maxmum possble number of purchases. We need to ensure γj [k + 1] >γ j [k]. γj [k] > 0 and x j [k] =0:hs mples that γ j [k] ˆx l j and the user s recevng more coupons than he can use. We need to ensure γj [k + 1] <γ j [k]. γj [k] > 0 and x j [k] > 0:hs mples that γ j [k] < ˆx l j and the user s not recevng enough coupons to realze the maxmum possble number of purchases. We need to ensure γj [k + 1] >γ j [k]. Note that an ncrease n the coupon rate cannot cause a decrease n the number of purchases. A smple controller that takes nto account all the four possble condtons s γ j[k + 1] = γ j[k]+δχ { γ j [k] z j [k]>0} +γ j[k] δχ { γ j [k] z j [k]=0}. 13 Here, δ> 0 s a constant small amount by whch we ncrease or decrease γj. We can now easly prove that the controller converges to wthn δ/2 of the desred value of ˆγ j. heorem 3: Under the tme scale separaton assumpton, usng the controller 13 we have lm k γ j[k] ˆx l j δ/2 S, j R. Proof: he proof s presented n Appendx C. Note that when δ s smaller enough and the algorthm starts wth a small γj [0], we can guarantee that γ j [k] b j for all k. Combnng Lemma 1, heorem 2 and heorem 3, we conclude that the number of coupons consumed under the two tme-scale algorthm converges to the optmal soluton to OP 1. VI. DELAY-BASED COUPON FORWARDING Suppose that the store rewards relays only after a coupon has been used to make a purchase. he nsght that we obtan from the optmalty of backpressure s the followng: If coupons are not used on a partcular path, queues buld up. hs would cause the average delay n beng rewarded to all relays on the path to ncrease. If a hgher rate of coupons than that set by the store are transferred along a path, the store does not reward the relays for some fracton of coupons and vrtual coupons buld up. Agan, ths would mean that the average delay n beng rewarded to all relays on the path would ncrease. he observaton mmedately suggests that perhaps a smpler algorthm would be to replace the backpressure based user control of Secton V A3 wth a much smpler scheme. Users need only keep track of the average delay experenced n obtanng rewards when they forward coupons to each of ther neghbors. hey choose to forward coupons to that neghbor who has the lowest such delay. he scheme s ntutvely ncentve compatble, snce users mght want to obtan rewards as soon as possble. hus, we may replace the reward scheme of Secton V A2 wth an equal reward for all users n the path. However, a few further addtons are requred to construct a workable heurstc algorthm. he frst addton stems from the fact that under backpressure, f a user fnds that all her neghbors have larger effectve queue lengths than herself, she does not forward coupons to any of them. he equvalent n the delay based regme would be to smply choose a threshold value of delay e.g., D U, and refuse to forward coupons to any neghbor that yelds a delay larger than ths threshold. he second addton s that whle keepng track of delays, even small dfferences n delays could result n a partcular user beng gnored entrely. Hence, nstead of a hard comparson between the delays of dfferent optons, we soften the comparson. For example, f neghbors 1 and 2 of a node yeld delays d 1 and d 2, we consder both as equally lucratve optons f d 1 d 2 D, where D s a constant delay threshold. Our expectaton s that ths smplfed algorthm would perform almost as well as the backpressure-scheme. Based on the observatons above, we propose the followng delay-based coupon forwardng to replace the user control of Secton V A3 for all coupons n whch user s not nterested. Delay-based coupon forwardng: Consder product j that user s not nterested. Denote by Dmt the average delay experenced n obtanng rewards when user j forwards type coupons to neghbor m. User keeps track Dmt for all neghbors. At tme step t, user j frst selects type coupon such that arg mn mn m:j,m L D mt and a subset of neghbors assocated wth type coupon { } Kj D = m : m t mn m:j,m L D m D. hen user j sends D m t DU,j,m L mn { Q j t,µ jt } K j number of type coupons to each of the neghbors n K j. Remark: Backpressure based user control requres a user to obtan the lengths of coupon queues from her/hs neghbors and from the store. Delay-based coupon forwardng, on the other hand, does not requre any nformaton exchange among the users. Each user makes decsons based on her/hs own nformaton hstory, whch results n a much smaller communcaton overhead as compared to backpressure based user control. Further, unlke backpressure, the reward gven to every user n the path of a coupon can be dentcal. VII. SIMULAION RESULS We smulate our coupon dstrbuton system on dfferent network topologes to study the valdty of our schemes. For the sake of comparson, we also create a thrd coupon dstrbuton system n whch coupons are forwarded by relays randomly to ther neghbors. hs would ndeed be the case f multhop coupon dstrbuton were allowed wthout ncentves

7 1 0.9 Error n xl Error n xh Error n xl Error n xh Error n xl Error n xh FRACIONAL ERRORS FRACIONAL ERROR FRACIONAL ERROR IME 0 IME 0 IME a Random Dstrbuton b Backpressure Dstrbuton c Delay-based Dstrbuton Fg. 2. Example trajectores of fractonal errors as compared to ˆx h 3 and ˆxl 3 n the number of goods bought by user 3 at the marked and dscounted prce. for forwardng n any partcular drecton. Intutvely, such a dstrbuton scheme should over-dstrbute coupons, snce the dstrbutor receves no feedback. A. Smple ree opology A smple tree topology s llustrated n Fgure 3. here s a sngle coupon source, two relays, sx leaf nodes customers, and one store. Relays may choose one of ther neghbors to forward coupons to at each tme nstant. Each customer j has a dfferent value of ˆx l j and ˆxh j. At each tme nstant, users utlze all the coupons that they possess f the cumulatve number of purchases made s less than ˆx l j +ˆxh j. Once ths s done, they purchase a random number of goods at the marked prce, as long as t s ratonal to do so.e., ether x l j [t] ˆxl j and x h j [t] ˆxh j, or xl j [t] +xh j [t] ˆxl j +ˆxh j and xh j [t] ˆxh j. Users repeat ths process untl the end of the small tme perod = 300. At the last nstant t = 1, f x h j [t] ˆxh j, user j purchases ˆx h j xh j [t] goods. Coupon Source NUMBER OF GOODS BOUGH Dscounted Prce Marked Prce IME Fg. 4. An example trajectory for user 3 over the small tme scale. he sold lne ndcates purchases made at the marked prce, whle the dashed lne ndcates dscounted purchases. he user has ˆx l j / = 50 and ˆxh j / = 60. In ths example we have set for llustraton γ j = ˆx l j / = 50,.e., the reward rate for coupon forwardng s known exactly, and we see that the user receves exactly the rght number of coupons Fg. 3. Store Smple tree topology. We frst verfy that the small tme scale dynamcs of backpressure s able to dstrbute the correct number of coupons to any user j. he capactes of all the relay lnks are set to 300 coupons per unt tme. We llustrate the trajectory of purchases made by user 3 who has ˆx l 3/ = 50 and ˆx h 3/ = 60 over a tme nterval = 300 unts n Fgure 4. For purposes of llustraton, we assume that γ 3 =ˆx l 3/ = 50 In other words, we set the reward rate for coupon forwardng by neghbors of user 3 exactly equal to the average rate at whch the user 3 should be gven coupons n order to extract maxmum revenue. We see n Fgure 4 that the backpressure scheme ndeed gves the rght number and rate of coupons to the user, ensurng that x l 3[ ]/ = 50 and x h 3[ ]/ = 60. We now smulate all three schemes small and large tme scales and the results are as follows. Fgure 2a shows the fractonal error n hgh and low prce purchases made by user 3, as compared to ˆx h 3 and ˆx l 3 for the delay based-scheme. We notce that there s a sgnfcant error. We plot the same quanttes when we use the backpressure-scheme n Fgure 2b. he scheme quckly converges, causng the errors to be small. Fnally, we plot the same for the delay-based scheme n Fgure 2c. For ths scheme, we chose the cut-off threshold to be /8 and the acceptable delay dfference to be 15%. Its error s between the other schemes. Fnally, we plot the total revenue obtaned by the store for the two dfferent schemes, and compare them to the maxmum possble revenue n Fgure 5. he upper bound s the value j pˆxh j + qˆxl j, whch s the maxmum extractable revenue. We see that the randomzed algorthm does sgnfcantly worse than backpressure as well as delay-based schemes. Even accountng for the fact that a constant part of the revenue would have to be used to ncentvze the scheme, the performance mprovement s stll sgnfcant, although the delay-based scheme performs worse than backpressure.

8 Fg. 5. REVENUE OBAINED Backpressure Max Random Delay Based 420 IME rajectory of revenue obtaned by the store usng dfferent schemes. B. Power Law opology We now consder a power-law topology that mght bear a closer resemblance to real-world socal networks. he graph conssts of 100 nodes, and s constructed usng preferentalattachment [18] wth each enterng node connectng to two others. Nodes are connected to the coupon source, are relays, and are customers wth probabltes 0.2, 0.7 and 0.1, respectvely. Customers have arbtrary spendng capactes. We show the upper bound and the performance of the three schemes n Fgure 6. Backpressure clearly performs the best, followed by the delay-based and random schemes. he results ndcate that usng such coupon dstrbuton schemes could sgnfcantly ncrease the revenue obtaned. REVENUE OBAINED Max Backpressure Random Delay Based 340 IME Fg. 6. rajectory of revenue obtaned by the store usng dfferent schemes for the power-law topology. VIII. CONCLUSION We developed dstrbuted schemes for targeted coupon delvery usng onlne socal networks. he objectve was to create a two-ter prce structure for maxmum revenue extracton by selectve dscountng. We desgned systems that allow users to obtan coupons from ther neghbors, and ncentvze these neghbors by rewardng them for coupon forwardng. We proved how backpressure deas could be used to acheve a optmal soluton, and also how to use t to obtan a smpler albet less effcent scheme. Future work ncludes dealng wth potental malcous users, and a testbed mplementaton. REFERENCES [1] Facebook, [2] Orkut, [3] mgnger, [4] L. assulas and A. Ephremdes, Stablty propertes of constraned queueng systems and schedulng polces for maxmum throughput n multhop rado networks, IEEE ransactons on Automatc Control, pp , December [5] M. Neely, E. Modano, and C. L, Farness and optmal stochastc control for heterogeneous networks, n Proc. IEEE Infocom., vol. 3, Mam, FL, March 2005, pp [6] A. Stolyar, Maxmzng queueng network utlty subject to stablty: Greedy prmal-dual algorthm, Queueng Systems, vol. 50, no. 4, pp , August [7] A. Erylmaz and R. Srkant, Jont Congeston Control, Routng and MAC for Stablty and Farness n Wreless Networks, IEEE Journal on Selected Areas n Communcatons, vol. 24, no. 8, pp , August [8] L. Chen, S. H. Low, M. Chang, and J. C. Doyle, Cross-layer congeston control, routng and schedulng desgn n ad hoc wreless networks, n IEEE Infocom, Barcelona, Span, Aprl [9] F. P. Kelly, Mult-armed bandts wth dscount factor near one: he Bernoull Case, Annals of Statstcs, vol. 9, pp , [10] P. Auer, N. Cesa-Banch, Y. Freund, and R. Schapre, he Nonstochastc Multarmed Bandt Problem, SIAM Journal on Computng, vol. 32, no. 1, pp , [11] N. Abe, A. Bermann, and P. Long, Renforcement learnng wth mmedate rewards and lnear hypotheses, Algorthmca, vol. 37, no. 4, pp , [12] M. Andrews, K. Kumaran, K. Ramanan, A. Stolyar, R. Vjayakumar, and P. Whtng, CDMA data QoS schedulng on the forward lnk wth varable channel condtons, Bell Labs ech. Memo, Aprl [13] X. Ln and N. Shroff, Jont rate control and schedulng n multhop wreless networks, n IEEE Conference on Decson and Control, Paradse Island, Bahamas, December [14] A. Erylmaz and R. Srkant, Far resource allocaton n wreless networks usng queue-length-based schedulng and congeston control, n Proc. IEEE Infocom., [15] E. Yeh and R. Berry, hroughput optmal control of wreless networks wth two-hop cooperatve relayng, n Proc. IEEE Int. Symp. Informaton heory ISI, June [16] L. Bu, R. Srkant, and A. L. Stolyar, Optmal resource allocaton for multcast flows n multhop wreless networks, he Phlosophcal ransactons of the Royal Socety, [17] L. assulas and A. Ephremdes, Dynamc server allocaton to parallel queues wth randomly varyng connectvty, IEEE rans. Inform. heory, vol. 39, pp , March [18] A.-L. Barabás and R. Albert, Emergence of scalng n random networks, Scence, vol. 286, pp , APPENDIX A: PROOF OF LEMMA 1 Frst, t s easy to verfy that the optmal soluton satsfes yj,s xl j because otherwse, the store can extract more proft by reducng the number of coupons sent to user j. Based on that, OP 1 can be re-wrtten as: OP1 : max q y h j,s + pˆx j S j B s.t. ym,j µ m, m N S j:m,j L,j s ym,j = yn,m m N j:m,j L n:n,m L y j,s xl j y m,j =0f m B and j s

9 Snce ˆx h j are constants, the objectve s equvalent to max q yj,s = max q y j,s. S S j B j B hus, the lemma holds. APPENDIX B: PROOF OF HEOREM 2 he analyss follows the Lyapunov drft used n [5], [6], [7]. Defne Q[t] ={Q j [t], Q j [t]} j N, S. It s easy to verfy that Q[t] s a Markov chan. Further, gven γj b j for all j and, we can obtan that for any j B, the followng holds = Q j[t + 1] + Q j[t + 1] Q j[t]+ Q j[t]+ m:m,j L ym,j [t] γ j Next, consder a Lyapunov functon such that V [t] = 1 Q 2 j[t]+ Q 2 j[t]. S j B Followng the analyss n [7], t can be shown that there exsts B>0, ndependent of Q[t], such that E [V [t + 1] V [t] Q[t]] B + Q d[t]e Θ [t] S a + S,j d j,m L S Q j[t]+ Q j[t] ˇy j,m j:d,j L + yd,j [t] Q[t]. E ym,j [t] yn,j [t] m:m,j L n:j,n L Q[t]. APPENDIX C: PROOF OF HEOREM 3 Lettng ˇΘ = We use a Lyapunov argument, wth the Lyapunov functon j:d,j L ˇy j,d, where ˇy j,d s the optmal J[k] = γ soluton to OP 2, we further obtan that j[k] ˆγ 2 j. hen we have E [V [t + 1] V [t] Q[t]] = B 1 + Q d [t] Q q E [ Θ [t] ˇΘ ] Q[t] J[k + 1] J[k] S + = γj[k + 1] 2 + ˆγ j 2 2ˆγ jγ j[k + 1] Q d[t] ˇΘ γj[k] 2 ˆγ j 2 + 2ˆγ jγ j[k] S yj,m [t] = γj[k + 1] γj[k] γj[k + 1] + γj[k] 2ˆγ j j m:j,m L S We have two cases. Q j[t]+ Q j[t] Q m[t] Q Case I: If γj m[t] x j [k] > 0,.e., γ j [k] < ˆγ j, we have from 13 J[k + 1] J[k] =δ2γ Next we have that j[k] ˆγ+δ, yj,m Q [t] j[t]+ Q j[t] Q m[t] Q whch s non-postve except n γj m[t] [k] ˆγ j [ δ/2, 0]. Case II: γj j,m L S x j [k] = 0,.e., γ j [k] ˆγ j, we have from 13 Q j[t]+ Q j[t] Q m[t] Q J[k + 1] J[k] = δ2γ m[t] j[k] ˆγ+δ, S Q d[t] ˇΘ, where nequalty a holds due to backpressure routng, and Q d [t] Q q Θ [t] ˇΘ 0 holds accordng to the defnton of the rate controller. hus, accordng to the Foster s crteron, we can conclude that Q[t] s postve recurrent, whch mples that lm t E[V [t]] <. Snce Q[t] s postve recurrent, we further have 1 E [V [ ]] E [V [0]] = 1 E [E[V [t] Q[t]]] E [E[V [t 1] Q[t 1]] t=1 B 1 + Q q t=0 Θ [t] whch mples that q ˇΘ t=0 Θ [t] B + Q Note that ˇΘ q ˇΘ t=0 Θ [t] 0, E [V [ ] V [0]]. because the network s stable and the ˇΘ s the optmal soluton to OP 2. hus, we have that 0 q ˇΘ t=0 lm Θ [t] B, Q whch leads to equalty 11. Furthermore, snce the queues are stable, so when, almost all coupons are consumed, whch mples that equalty 12. whch s non-postve except n γj [k] ˆγ j [0, δ/2]. hus, the system s globally asymptotcally stable and γj wll converge to the nterval [ δ/2, +δ/2]. ˆγ j