Bilateral Single-Issue Negotiation Model Considering Nonlinear Utility and Time Constraint

Transcription

1 Bilateral Single-Issue Negotiation Model Considering Nonlinear Utility and Time Constraint Fenghui Ren School of Computer Science and Software Engineering University of Wollongong, Australia Minjie Zhang School of Computer Science and Software Engineering University of Wollongong, Australia Abstract Bilateral agent negotiation is considered as a fundamental research issue in autonomous agent negotiation, and was studied well by researchers. Generally, a predefined negotiation decision function and utility function are used to generate an offer in each negotiation round according to an negotiator s negotiation strategy, preference, and restrictions. However, such a negotiation procedure may not work well when the negotiator s utility function is nonlinear, and the unique offer has difficult to be generated. That is because if the negotiator s utility function is non-monotonic, the negotiator may find several offers that come with the same utility at the same time; and if the negotiator s utility function is discrete, the negotiator may not find an offer to satisfy its expected utility exactly. In order to solve such a problem, we propose a novel negotiation model in this paper. Firstly, a 3D model is introduced to illustrate the relationships between an agent s utility function, negotiation decision function and offer generation function. Then two negotiation mechanisms are proposed to handle two types of nonlinear utility functions respectively, ie. a multiple offers mechanism is introduced to handle non-monotonic utility functions, and an approximating offer mechanism is introduced to handle discrete utility functions. Lastly, a combined negotiation mechanism is proposed to handle nonlinear utility functions in general situations by considering both non-monotonic and discrete. The experimental results demonstrate the effectiveness and efficiency of the proposed negotiation model. Keywords: Agent negotiation, nonlinear utility function, negotiation decision function, offer generation function addresses: fren@uow.edu.au (Fenghui Ren), minjie@uow.edu.au (Minjie Zhang) Preprint submitted to Decision Support System August 1, 2012

2 Figure 1: An employee s temper for a day. 1. Introduction Agent negotiation is one of the most significant research issues in multi-agent systems (MASs). Many works have been done to solve challenges in agent negotiation. To list a few of them, Narayanan et al. [1, 2] adopted a Markov chain framework to model bilateral negotiation and employed Bayesian learning to enable agents to learn an optimal strategy in incomplete information settings. Fatima et al. [3] investigated the negotiation outcomes in incomplete information settings through the comparison of the difference between two agent s negotiation deadlines, and proposed an agenda-based framework to help selfinterested agents to maximize their utilities. Brzostowski et al. [4] proposed an approach to predict the opponent s behaviors based only on the historical offers of the current negotiation. They claimed that time and imitation are two main factors which influence an agent s behaviors during negotiation. Ren et al. [5] proposed a market-based model to handle the uncertainty and concurrency in open and dynamic negotiation environments. However, most existing approaches are based on the assumption that all negotiators employ monotonic continuous linear utility functions, and not much work has been done on negotiations in which agents employ non-monotonic and/or discrete nonlinear utility functions. According to our studies, agents may employ such nonlinear utility functions in many real-world negotiations [6][7]. For example, as shown in Figure 1, in a scheduling problem for task allocation, an employee feels happy to be assigned work between 9AM 12AM and 1P M 3P M, but feels unhappy to work hard during the first one or last two hours of a day. The employee s temper in a working day is a non-monotonic function. In Figure 2, a potential car purchaser may have different preferences on a car s color. Because each model of a car only has limited colors, the car purchaser s preferences on a car s color is a discrete function. In a negotiation with time constraint, an agent usually defines a negotiation 2

3 Figure 2: A customer s flavour on a car s color. strategy to make concessions throughout a negotiation. Firstly, according to the negotiation decision function, agents can calculate the possible maximal utility they can gain at a certain moment. Then, according to the offer generation function, agents can find a particular offer to reach their expected utility. Because most negotiation models assume that agents employ monotonic and continuous utility functions, a particular offer can always be found to satisfy agents expected utilities at any negotiation round. Also, it can be guaranteed that the concessions from opponent s offers are always consistent, i.e., always monotonically increasing or decreasing the agents utilities. However, when agents employ non-monotonic and/or discrete utility functions, it cannot be guaranteed that the opponent s offers are always in a ascending or descending order. Also, the particular offer which can exactly match the expected utility may not be found. Therefore, when the utility function is non-monotonic, agents may have multiple options on offers in order to reach the expected utility. As shown in Figure 1, in order to ensure that an employee s happiness is p (p [0, 1]), a job can be assigned to the employee at either 9AM or 12AM. Also, when the utility function is discrete, an agent perhaps cannot find an offer to satisfy the expected utility exactly. As shown in Figure 2, none of the available colors can make a car purchaser s happiness equal to q (q [0, 1]) exactly. In order to solve the offer generation problem when agents employ nonlinear utility functions, we propose a novel negotiation model in this paper, which can handle the both situations when agents have non-monotonic and/or discrete utility functions. Specifically, for negotiations involving non-monotonic utility functions, the multiple offers mechanism is introduced to allow agents to generate equivalent offers in a negotiation round; for negotiations involving discrete utility functions, the approximating offer mechanism is introduced to allow agents to generate an offer to approximate their expected utilities. Eventually, the two mechanisms are combined to solve general situations in negotiations involving nonlinear utility functions. Furthermore, when utility functions are non-monotonic, the existing alternating offers protocol [8] may become ineffi- 3

4 cient in enlarging agents profits. That is because the existing alternating offers protocol assumes an agent has a consistent evaluation on offers changes, i.e., when an agent gradually makes concessions to decrease own profit, the opponent s profit will be gradually increased, and vice versa. However, such an assumption is not held when agents employ non-monotonic utility functions. In order to help agents to make decisions on trade off in such a situation, a new negotiation protocol is proposed based on the alternating offers protocol. The rest of this paper is organized as follows. Section 2 briefly introduces a general bilateral single issue negotiation model with linear utility functions. Section 3 introduces our 3D negotiation model, the multiple offers mechanism to handle non-monotonic utility functions, the approximating offer mechanism to handle discrete utility functions, and the combined mechanism to handle general nonlinear utility functions. In Section 4, the Rubinstein s alternating offers protocol is modified to fit the situation caused by non-monotonic and/or discrete utility function. Section 5 demonstrates a negotiation between two agents having nonlinear utility functions. Section 6 compares our work with some related work on handling negotiations with nonlinear utility functions. Section 7 concludes this paper and explores our future work. 2. A General Negotiation Model for Linear Utility Function Before we introduce the negotiation model for nonlinear utility functions, we would like to briefly introduce a general negotiation model for linear utility functions by considering time constraints. A general bilateral single issue negotiation is performed between two agents on a good s price. Let b denote the buyer, s denote the seller, and let [IP a, RP a ] denote the range of values for price that are acceptable to Agent a, where a {b, s}. IP a is Agent a s initial price and RP a is Agent a s reservation price. Usually, for a buyer agent IP b RP b, and for a seller agent IP s RP a. We use ˆa to denote Agent a s opponent, ˆa {b, s}. Obviously, an agreement can be reached between Agent a and its opponent ˆa only when there is an intersection between their price ranges. If the buyer agent s reservation price is smaller than the seller agent s reserved price (i.e., RP b < RP s ), an agreement will not be achieved. Usually, a negotiation should consider time constraint, and each agent has a negotiation deadline. If an agreement cannot be achieved before an agent s deadline, then the agent has to quit the negotiation, and the negotiation fails. Let T a denote Agent a s deadline. A negotiation can be started by either the buyer or seller. During the negotiation, the buyer and the seller will send alternating offers to each other until both sides agree on an offer together, or one side quits the negotiation. This negotiation protocol is known as the alternating offers protocol [8]. Let ptˆa a denote the price sent from Agent ˆa to Agent a at time t. Once Agent a receives the offer, it will map the price in the offer to a utility value by using its utility function U a. A general linear utility function used by Agent a is shown in Equation 1. U a (p tˆa a) = 4 ptˆa a RP a IP a RP a (1)

5 It can be seen that a general linear utility function normalizes a price to a utility value in-between [0, 1] by using the predefined initial price (IP a ) and reservation price (RP a ). The utility value indicates the profit that the agent can gain by accepting the offer p tˆa a. In general, for Agent a, if U a (p tˆa a ) is greater than the value of the counter-offer Agent a is ready to send in the next negotiation round t, i.e., U a (p tˆa a ) U a (p t a ˆa ), then Agent a will accept Agent ˆa s offer at round t and the negotiation completes successfully with the agreement ptˆa a. Otherwise, the counter-offer pt a ˆa will be sent from Agent a to Agent ˆa. Such a procedure will be repeated until an agreement is achieved or one agent reaches its deadline. Thus, the action, A a, that Agent a takes at each negotiation round t is usually defined as follows: A a (p tˆa a) Quit if t > T a, = Accept p tˆa a if U a (p tˆa a ) U a (p t a ˆa ), (2) Offer p t a ˆa otherwise. If Agent a does not accept the price ptˆa a and its deadline does not achieve, then it will send a counter-offer p t a ˆa to Agent ˆa as a response. Usually, agents may employ different negotiation tactics [9] to generate counter-offers based on different criteria, such as time, resources, and previous counter-offers. The time-dependent tactic is the most popular criteria when agents generate their counter-offers by considering the time constraint. In the time-dependent tactic, the predominant factor used to decide which price should be offered in round t, and vary the offer by considering the round t and the deadline T a. Usually, the counter-offer made by Agent a at round t (0 t T a ) is modeled as a function Φ a as follows: p t a ˆa = { IP a + Φ a (t)(rp a IP a ) RP a + (1 Φ a (t))(ip a RP a ) for a = b, for a = s. where function Φ a (t) is called the Negotiation Decision Function (NDF) [9] and is defined as follows: (3) Φ a (t) = k a + (1 k a )( t T a )1/ψ (4) where k a is the initial utility and k a [0, 1]. If k a = 0, then at the beginning of a negotiation the initial price IP a will be selected, and when the deadline is reached the reserved price RP a will be selected. Theoretically, the NDF has an infinite number of possible tactics when ψ (ψ 0) has different values. However, in Figure 3 three extreme cases show different patterns of behaviors when ψ located in different ranges. Conceder: When ψ > 1, the rate of change in the slope is decreasing. It represents a negotiation behavior where the agent will give larger concessions during the early stages of the negotiation, but smaller concessions during the later stages of the negotiation. 5

6 Figure 3: Negotiation decision functions for the buyer [3]. Linear: When ψ = 1, the rate of change in the slope is zero. It represents a negotiation behavior where the agent will give a constant concession throughout the negotiation. Boulware: When 0 < ψ < 1, the rate of change in the slope is increasing. It represents a negotiation behavior where the agent will give smaller concession during the early stages of the negotiation, but larger concessions during the later stages of the negotiation. In this section, we briefly introduced a general bilateral single-issue negotiation model. However, such a model is based on an the assumption that all agents employ linear utility functions, i.e., Equation 1 is monotonic and continuous. Therefore, during a negotiation, an unique offer can be generated to reach an agent s utility indicated by the negotiation decision function (see Equation 4). However, if the utility functions employed by agents are non-monotonic and/or discrete, this general model cannot guarantee to produce correct offer/s to reach the expected utility, and agents reactions (see Equation 2) during negotiations may become inefficient in enlarging agents utilities. In the following sections, we will introduce several negotiation mechanisms to handle the situation when agents employ non-monotonic and/or discrete utility functions. 3. Offer Generation Considering Non-monotonic Utility Function When agents employ non-monotonic utility functions, the relationships between offers and utilities are not one-to-one mapping anymore, but more than one offers may contribute to the same utility. In this subsection, we firstly propose a multiple offers mechanism to handle such a problem when agents employ non-monotonic utility functions. 6

7 Dimensional Model In order to well illustrate the relationship among an agent s negotiation decision function, utility function, and counter-offer generation function, we proposed a 3D model. As displayed in Figure 4, the front surface (Surface A) is the negotiation strategy surface (NS surface). It illustrates the relationship between the negotiation round and the agent s expected utility. Let the x-axis indicate the negotiation round (t) and z-axis indicate the agent s utility (u), then the mapping displayed on NS surface is summarized in Equation 5. So the negotiation decision function introduced in Equation 4 (see Section 2) belongs to this class. φ a t = Φ a (t) (5) The side surface (Surface B) is the utility function surface (UF surface). It illustrates the relationship between the real value of the negotiating issue and the agent s utility. Let y-axis indicate the real value of the negotiating issue, then the mapping displayed on UF surface could be summarized in Equation 6. u a o = U a (o) (6) where o indicates an offer, U a ( ) is Agent a s utility function, and u a o is Agent a s evaluation on offer o. The bottom surface (Surface C) is the offer generation surface (OG surface). It illustrates the relationship between the negotiation round and the agent s offers. In Equation 7, we summarized such a relationship. o a t = Ψ a (t) (7) where Ψ a (t) is Agent a s offer generation function, and o a t is the offer generated by Agent a at round t. During a negotiation, agents should have such an offer generation function to create a counter-offer in each negotiation round once the opponent s offer is not acceptable. For example, the Equation 3 defines a general offer generation function by considering the initial offer, reservation offer and time constraint. However, Equation 3 is based on the assumption that the agent employs a linear utility function, so an unique offer can be selected in each negotiation round to fit the expected utility. When agents employ non-monotonic and/or discrete utility, the one-to-one mapping between the negotiation round and the offer may not exist anymore. In the following subsections, three mechanisms will be introduced to solve such a problem Multiple Offers Mechanism In this subsection, we propose a multiple offers mechanism to solve the offer generation problem when agents employ non-monotonic utility functions. Let U a (o) be the non-monotonic utility function that Agent a employs, and Φ a (t) be Agent a s decision making function by considering time constraint, then the problem is how to create the offer generate function Ψ a (t) by using U a (o) and Φ a (t). 7

8 Figure 4: Negotiation model for non-monotonic continuous utility function. Let φ a t be Agent a s expected utility at round t, U ( 1,a) ( ) be the inverse function of the non-monotonic utility function U a (o), then the offers (O) which contribute equivalent profit as the expected utility φ a t can be calculated as follows. O a t = U ( 1,a) (φ a t ) (8) where φ a t is decided by the negotiation decision function Φ a (t), and O a t contains the offers that Agent a should send to the opponent at round t. Because the utility function U a (o) is not a monotonic function, so the inverse function U ( 1,a) (φ a t ) is not an one-to-one mapping function and may generate more than one counter-offers for the same expected utility. Finally, Agent a s offer generation function by considering time constraint is defined as follows. Ψ a (t) = U ( 1,a) (Φ a (t)) (9) where Φ a (t) is Agent a s negotiation decision function, and a general Φ a (t) is defined in Equation 4. As shown in Figure 4, Curve X on Plane A is the negotiation decision function, Curve Y on Plane B is the utility function. Both the negotiation decision function and the utility function are known by Agent a. The proposed 3D model indicates well how to produce the offer generation function (Curves Z and K on Plane C) based on these two known functions when the utility function is non-monotonic. For example, at the negotiation round t, according to the negotiation decision function, Agent a expects a utility of 0.8, and according to 8

9 the utility function (Line Y ), two offers can reach the expected utility (Points m and n). Then, by employing Equation 8, the two offers (offers b and p) are found by Agent a. Finally, Agent a generates two counter-offers (Points e and f) at round t to its opponent. By employing such a multiple offer mechanism at each negotiation round, Agent a s counter-offer generation function can be drawn on Surface C. It can be seen that Agent a s counter-offers during the negotiation are illustrated by curves Z and K. Based on the above description, in order to create the offer generation function when agents employ the non-monotonic utility function, the following three steps are highlighted. The detailed algorithm is given in Algorithm 1. Step 1: According to an agent s negotiation decision function, the agent calculates its expected utility at the negotiation round t; Step 2: According to the agent s non-monotonic utility function, the agent discovers value/s on the negotiation issue. From these discovers values, the agent should gain exact same profit as its expected utility at the round t; Step 3: The agent makes a decision whether to accept the opponent s offer or to send the counter-offer (see Section 4 for detail). Once the counter-offer needs to be sent, all the equivalent offers are sent at the same time as a package to the opponent. Algorithm 1 Offer generation when agents employ non-monotonic utility function Input: Agent a s negotiation decision function Φ a (t), utility function U a (o) and all possible offers O Output: Agent a s counter-offer/s O c at the negotiation round t O c, φ a t Φ a (t) O t U ( 1,a) (φ a t ) for all o t,i O t do if o t,i O then O c {o t,i } O c end if end for return O c As shown in Algorithm 1, in order to generate the multiple offers, the solutions of the agent s inverse utility function should be calculated firstly, and then the solutions should be compared with the offer candidate set for validity checking. Therefore, the complexity of the offer generation, when the non-monotonic utility function is used, is T (n) = O(U ( 1,a) ) + 2n

10 Figure 5: Negotiation model for discrete utility function Approximating Offer Mechanism In some situations, agents may employ discrete utility functions. Because the number of offers is limited when agents use discrete functions, the agents may not find offers to satisfy their expected utilities exactly. In order to handle such a problem in discrete utility functions, we propose an approximating offer mechanism in this subsection. Let set O = {o i i [1, I]} be the all possible offers that Agent a can choose during a negotiation, and notation u a o i be the utility of Offer o i. At the negotiation round t, Agent a firstly employs the negotiation decision function to calculate its expected utility, i.e., φ a t (see Equation 4). The value of φ a t indicates the maximal utility that Agent a may gain in the remaining negotiation rounds. Because Agent a employs a discrete utility function, a particular offer which can generate the exact value as the expected utility may not exist. Therefore, Agent a should select another offer which can generate a utility to approximate the expected utility as much as possible. Let d a t,o i indicate the distance between the expected utility φ a t and the offer o i s utility u a o i for Agent a at round t. Intuitively, d a t,o i should be greater than 0, because φ a t is Agent a s best expectation at round t. So d a t,o i is defined as follows: d a t,o i = φ a t u a o i (10) Then, the counter-offer that Agent a should select is indicated by notation o c, and o c is decided as follows. o c = o i O, o c O d a t,o c < d a t,o i d a t,o c 0 (11) 10

11 Algorithm 2 Offer generation when agents employ discrete utility function Input: Agent a s negotiation decision function Φ a (t), utility function U a (o) and all possible offers O Output: Agent a s counter-offer o c at the negotiation round t O t, φ a t Φ a (t), d min + for all o i O do u oi U a (o i ) d a t,o i φ a t u a o i if d a t,o i 0 then O t {o i } O t if d a t,o i < d min then d min d a t,o i end if end if end for for all o t,i O t do u a o t,i U a (o t,i ) d a t,o t,i φ a t u a o t,i if d a t,o t,i == d min then o c d a t,o t,i return o c end if end for Equation 11 represents an approximating approach for counter-offer generation when agents employ discrete utility functions. It is guaranteed that the offer which can mostly satisfy an agent s expected utility will be found from its candidate offers. In Figure 5, we illustrate an example to show the process in detail. Curve X is an agent s negotiation decision function, and located on Plane A. The agent s possible offers are distributed discretely on Plane B, and indicated by triangles. And the agent s offer generation function is Curve Z on Plane C. Because the agent s utility function is a discrete function, so the offer generation function is a piecewise function. For example, as shown in Figure 5, at round t, the agent s expected utility is 0.8. However, none of the agent s offers can bring 0.8 utility to the agent exactly. Then, by employing the approximating offer mechanism, Point h is found as the closest utility to 0.8, and Offer q is selected as the counter-offer. Finally, the agent sends the counter-offer q at round t, which is indicated by Point d on Curve Z. Based on the above description, the offer generation algorithm by considering the discrete feature of utility functions is defined in Algorithm 2, and the three major steps is summarized as follows. Step 1: According to an agent s negotiation decision function, the agent cal- 11

12 Figure 6: Negotiation model for general nonlinear utility function. culates its expected utility at the negotiation round t; Step 2: According to the agent s discrete utility function, the agent finds the offer which can reach the expected utility as much as possible from its offer candidates; Step 3: The agent makes a decision whether to accept the opponent s offer (see Section 4 for detail), and send the counter-offer in necessary. As shown in Algorithm 2, when the agent employs discrete utility function, the counter-offer generation approach does not use the inverse utility function, so as to greatly decrease the complexity. Therefore, the complexity of the offer generation, when the discrete utility function is used, is T (n) = 17n Combined Mechanism Subsection 3.2 proposed a multiple offers mechanism for agents employing non-monotonic utility functions, and subsection 3.3 proposed an approximating offer mechanism for agents employing discrete utility functions. In a more general situation, agents utility functions may be a mixture of non-monotonic and discrete functions. For example, as shown in Figure 6, an agent employs a non-monotonic and discrete utility function. At round t, the agent s expected utility is indicated by Point o, i.e., However, according to the agent s utility function, none of the agent s offers can satisfy its expectation exactly. So the agent should employ 12

13 the approximating offer mechanism to find the closest offer/s to its expectation. Because the agent s utility function is non-monotonic, so two points (i.e., Points m and n) have the similar distance to the agent s expected utility are found. So by employing the multiple offers mechanism, two counter-offers (Offers e and f) are sent at negotiation round t. Algorithm 3 Offer Generation When Agents employ non-monotonic and discrete utility function Input: Agent a s negotiation decision function Φ a (t), utility function U a (o) and all possible offers O Output: Agent a s decision at the negotiation round t O c, φ a t Φ a (t) O t U ( 1,a) (φ a t ) for all o t,i O t do if o t,i O then O c {o t,i } O c end if end for if O c == then O t, d min + for all o i O do u oi U a (o i ) d a t,o i φ a t u a o i if d a t,o i 0 then O t {o i } O t if d a t,o i < d min then d min d a t,o i end if end if end for for all o t,i O t do u a o t,i U a (o t,i ) d a t,o t,i φ a t u a o t,i if d a t,o t,i == d min then O c {o t,i } O c end if end for end if return O c Let set O = {o i i [1, I]} be all possible offers that Agent a can choose during a negotiation, set O c = {o c j j [1, J]} contains Agent a s all possible counter-offers at round t, and O c O. Then the counter-offer o c j is selected as 13

14 follows. o c j = o i O, o c j O d a t,o c j < da t,o i d a t,o c j 0 (12) and o c k O c, o c j O c, k = j d a t,o c k = da t,o c j (13) where function d a t,o i indicates the distance between Agent a s expected utility φ a t and the offer o i s utility u a o i at the negotiation t. Based on the above description, the offer generation function by considering both the non-monotonic and discrete utility function is given in Algorithm??, and the three major steps are given below as well. Step 1: According to an agent s negotiation decision function, the agent calculates its expected utility at the negotiation round t; Step 2: According to the agent s nonlinear utility function, firstly the agent tries to find the offers which can bring the exact same profit as its expected utility. If such offers cannot be found, then the agent tries to find the offer/s which can reach the expected utility as much as possible from its offer candidates; Step 3: The agent makes a decision whether to accept the opponent s offer (see Section 4 for detail), and send the counter-offer/s in necessary. As shown in Algorithm 3, when the agent employs the non-monotonic and discrete utility function, the complexity of counter-offer generation is the combination of Algorithms 1 and 2. Therefore, the complexity for Algorithm 3 is T (n) = O(U ( 1,a) ) + 12n Negotiation Protocol Considering Nonlinear Utility Function In this subsection, a negotiation protocol for agents with nonlinear utility functions is proposed based on Rubinstein s alternating offers protocol [9]. In Section 2, it introduced how an agent make a decision on whether the counteroffer should be sent to the opponent by considering time constraint. The basic idea is that Agent a compares the utility gained from the opponent ˆa s offer at the negotiation round t (U a (ptˆa a )) and the maximum utility the agent may gain in the following negotiation round t (U a (p t a ˆa )). If Agent a has the chance to gain more profit in the round t, i.e., U a (p t a ˆa ) > U a (ptˆa a ), then Agent a will reject opponent ˆa s offer ptˆa a and send the counter-offer pt a ˆa. Otherwise, Agent a will accept opponent ˆa s offer and gain U a (ptˆa a ) utility. However, such a decision making procedure is based on an assumption that all negotiation agents employ the linear utility functions, and may can not maximize agents profits when the non-monotonic utility functions are employed. In this section, we extends the Rubinstein s alternating offers protocol to make it suitable in negotiations when non-monotonic utility functions are employed by agents. Basically, agents may have three typical actions in each negotiation, 14

15 i.e., quit the negotiation, accept the opponent s offer, and send the counter-offer. Equation 14 indicates how an agent selects its actions in each negotiation round by considering negotiation round, opponent s offer, and its counter-offer. Quit, when t = τ a max(u a (p ṱ a a )) < 0, Act a (t) = Accept p t ˆa a, when t τ a U a (p t ˆa a ) Φa (t + 1), (14) Offer p t+1 a â, when t + 1 τ a U a (p t ˆa a ) < Φa (t + 1). where τ a is the Agent a s negotiation deadline, U a ( ) is the Agent a s utility function, set p ṱ a a contains all offers from the opponent ˆa to the Agent a, p t ˆa a is the best offer from the opponent and is defined in Equation 15. p t+1 a â p t ˆa a = p tˆa a p ṱ a a, p t ˆa a U a (p t ˆa a) U a (p tˆa a) (15) contains all counter-offer/s from the Agent a to the opponent ˆa at round t + 1, and are calculated by the offer generation algorithm 3. The detail action selection procedure is introduced in Algorithm 4. Algorithm 4 Negotiation protocol considering nonlinear utility function Input: Agent a s utility function U a ( ), negotiation decision function Φ a (t), offer generation function Ψ a (t), negotiation deadline τ a, and the offer/s from opponent ˆa at round t (p ṱ a a ). Output: Agent a s counter-action at the negotiation round t. u t max 0, p t ˆa a nil for all p tˆa a pṱ a a do u t U a (p tˆa a ) if u t > u t max then u t max u t, p t ˆa a ptˆa a end if end for φ a t+1 Φ a (t + 1) if t < τ then if u t max φ a t+1 then return Accept p t ˆa a else p t+1 a â Ψa (t + 1) return Reject and send p t+1 a â end if else if u t max φ a t+1 then return Accept p t ˆa a else return Quit end if 15

16 Based on the above description, the extended negotiation protocol when agents employ nonlinear utility functions are summarized as follows: Step 1: An agent assigns negotiation parameters before a negotiation starts, i.e., the initial offer, reservation offer, utility function, negotiation deadline, and negotiation decision function. Step 2: The agent calculates its best offer according to the offer generation approach introduced in Section 3 by considering time constraints, and sends offer/s to its opponent. Step 3: If the opponent accepts any offer, then the negotiation is completed. Otherwise, the opponent will send back the counter-offer/s, and wait for the agent s response. If the current negotiation round is the agent s deadline, then the procedure goes to Step 4. Otherwise, the procedure goes to Step 5. Step 4: The agent evaluates the opponent s counter-offer/s. If the opponent s counter-offer/s can bring any profit to the agent, then the agent will accept the opponent s counter-offer, and the negotiation succeeds with an agreement. If none of the opponent s counter-offer/s can bring any benefit to the agent, then the agent will reject all the opponent s offer/s, and the negotiation fails. Step 5: The agent calculates its offer for the next negotiation round according to the offer generation approach introduced in Section 3. If the opponent s counter-offer can briny more profit to the agent than the agent s offer for the next round, then the agent will accept the opponent s offer, and the negotiation succeeds with an agreement. If none of the opponent s counteroffer/s can exceed the agent s offer for the next round, then the agent will send the offers to the opponent, and the negotiation procedure goes back to Step Experiment In this section, we demonstrate the negotiation procedure by employing the proposed negotiation approach between two agents with nonlinear utility functions Setting We simulate the negotiation between a female patient and her dentist on appointment making. Both the patient and the dentist have their own preferences on schedule, and their preferences distributes nonlinearly in a day. Briefly speaking, a female patient wants to make an appointment with her dentist. The patient usually needs to deliver her children to school in the morning (9 : 00-9 : 30) and picks up them from school in the afternoon (16 : : 30). So she prefers not to make an appointment during school hours. However, if her 16

17 Figure 7: The patient s preference on her timetable. Figure 8: The dentist s preference on his timetable. dentist s timetable is full, and she has no other options, she will also accept an appointment during the school hours, and ask her neighbor to give a lift to her children. Also, she prefers to avoid lunch time (12 : 30-1 : 30), and the time during her favourite TV program (14 : : 30). Therefore, this patient s preference for an appointment time is a nonlinear function, which is illustrated in Figure 7. During the negotiation, the patient is represented by Agent p. On the other hand, the dentist s working hours are between 8 : 00 and 18 : 00. But he tries to avoid appointments in the early morning (8 : 00-9 : 00), lunch time (13 : : 00), and late afternoon (17 : : 00). Also, he already had two appointments at 10 : : 00 and 14 : : 00. Therefore, the dentist s preference for the appointment time is a nonlinear and discrete function, which is illustrated in Figure 8. During the negotiation, the dentist is represented by Agent d. In order to simply the negotiation procedure, both Agents p and d employ 17

18 Figure 9: The comparison between the dentist s utilities and the patient s utilities. linear negotiation decision functions (the correctness of the proposed mechanism is independent on the selection of the negotiation decision function), and set their negotiation deadlines to the 10th round. By employing the proposed multiple offers mechanism and the approximating offer mechanism, Agents p and d can efficiently generate and exchange their offers during the negotiation. Both the patient agent s and the dentist agent s offers in each negotiation round are generated and displayed in Figures 10 and 11, respectively. It can be seen that because both agents utility functions are non-monotonic, they send multiple offers in each negotiation round. The detailed negotiation procedure is displayed in Figure 9. In Figure 9, the x-axis indicates the patient agent s utility, and the y-axis indicates the dentist agent s utility. The solid line indicates the dentist agent s offers in each round, and the negotiation round is marked by the Greek numerals (I, II, III,...). The broken line indicates the patient agent s offers, and the negotiation round is marked by the Roman numerals (1, 2, 3,...). According to the patient agent s offer generation function (Figure 10), the patient agent sends two offers (i.e., 11 : 00 and 15 : 00) in the first negotiation round. These two offers bring the same utility (i.e., utility equals 1.0) to the patient agent, but different utilities to the dentist agent. According to the dentist agent s utility function (Figure 8), the first offer (11 : 00) can bring him a utility value as 0.6, while the utility value of the second offer (15 : 00) is zero. However, since the both offers from the patient agent failed in reaching the dentist agent s expectation, the dentist agent rejected them, and sent counter-offers, i.e., 10 : 00 18

19 Figure 10: The patient s offers in each negotiation round. Figure 11: The dentist s offers in each negotiation round. and 16 : 00. Both the counter-offers can bring 0.8 utility to the dentist and 0.6 utility to the patient agent. Because the patient s expected utility in the following negotiation round is 0.9, so the patient rejects the dentist s counteroffers, then the negotiation goes into the second round. In the second round, the patient agent sent four offers together (i.e., 10 : 45, 11 : 15, 14 : 45 and 15 : 15) to the dentist agent. Except the offer 11 : 15 takes utility to the dentist agent, the other three offers s utilities for the dentist are 0. Because is smaller than the dentist agent s expectation in the second round (i.e., 0.9 utility), the dentist agent rejected the four offers and still sent 10 : 00 and 16 : 00 as counter-offers to the patient agent. It should be mentioned that the dentist agent sent the same counter-offers as in the first round because the dentist s utility function is discrete and these two counter-offers maximized the dentist s utility in the first three round. Because the patient agent s utility function is still nonlinear, even though the dentist agent s counter-offers are 19

20 rejected by the patient agent in the first negotiation round, they still may be accepted by the patient in the second round. Therefore, in a negotiation when nonlinear utility functions are considered, agents should be allowed to send the same offers to the opponents in different negotiation rounds. In this experiment, the patient agent rejected these two counter-offers again because their utility are smaller than the patient s expected utility in the third round, i.e., 0.8 utility. Therefore, the negotiation goes into the third round. In the third round, the patient agent sent another four offers to the dentist agent, i.e., 10 : 30, 11 : 30, 14 : 30 and 15 : 30. Only the offer 11 : 30 can bring 0.55 utility to the dentist agent, and the dentist can not get any utility from the other three offers. However, because 0.55 is still smaller than the dentist s expected utility (i.e., 0.8) in the this round, the dentist agent denied the all four offers and send the similar counter-offers as it did in the first two negotiation rounds. Again, the dentist agent s counter-offers (i.e., 10 : 00 and 16 : 00) can only bring 0.6 utility, and fails to reach the patient s expected utility (i.e., 0.7) in the fourth negotiation round. Therefore, the patient agent rejects the dentist agent s two counter-offers, and the negotiation process to the fourth round. In the fourth round, the patient agent modified its offers and send them to the dentist agent, i.e., 10 : 15, 11 : 45, 14 : 15 and 15 : 45. These offers bring 0.7 utility to the patient, and maximally bring utility to the dentist. Because the dentist agent s expected utility in this round is 0.6, so the dentist agent rejected these four offers, and sent two counter-offers (i.e., 9 : 30 and 16 : 30). The dentist agent s counter-offers can only bring 0.35 utility to the patient agent, which is smaller than the patient agent s expected utility (i.e., 0.6) in the fifth negotiation round. Therefore, the dentist agent rejected these two counter-offers, and the negotiation process to the fifth round. Finally, the agreement was reached at the 5th negotiation round. In the 5th negotiation round, the patient agent sent four equivalent offers to the dentist agent, i.e., 10 : 00, 12 : 00, 14 : 00 and 16 : 00. All these four equivalent offers bring the same utility (0.6) to the patient agent, but different utilities to the dentist agent. According to the dentist agent s utility function (see Figure 8), two of the patient agent s offers (10 : 00 and 16 : 00) brought the utility value 0.8 to the dentist agent. Because the dentist agent s expected utility in the 5th round is only 0.6, so the dentist agent would take either one of these two offers as the agreement. The detail negotiation process is illustrated in Figure 9, and each agent s offers and evaluations are listed in Table 2. We also compare our proposed approach with two related approaches in literature. Firstly, we employed Ito et al. s approach [10] on the above scheduling problem. According to the Ito s approach, we firstly select ten samplings for both the dentist and the patient. The samplings are displayed in Table 1. Then a mediator agent was employed to explore the table to search for the sampling candidates which could bring the maximal utility by considering both dentist and patient. Finally, the offer 11 : 30 was selected as the agreement because it bring the largest profit (i.e., 0.8 for the patient, 0.55 for the dentist, and 1.35 as total) by considering both dentist and patient. By comparing with the agreements (i.e., 10 : 00 and 16 : 00) achieved by using our proposed approach, the 20

21 Table 1: Ten samplings for the dentist and the patient Patient Dentist Offer Utility Offer Utility 8 : : : : : : : : : : : : : : : : : : : : agreement achieved by using Ito s approach brought higher profit to the patient (0.8 > 0.6), but lower profits to the dentist (0.55 < 0.8). Therefore, the overall profit generated by our approach is better than Ito s approach, i.e., 1.4 > However, Ito s approach is greatly impacted by the selection of samplings. If suitable samplings could be found, Ito s approach could also theoretically reach the same agreement as our approach. Nevertheless, the selection of suitable samplings and non-bias mediator agent are not easy in real-world applications, and our approach avoids these two procedures. In the second comparison, we employed the approach proposed by Fatima et al. [11], i.e., to approximate the dentist and the patient s nonlinear utility functions to linear utility functions, then use the well-known NDF approach [9] to find the agreement. For the patient, the approximated linear utility function is U(x) = 0.09x ; and for the dentist, the approximated linear utility function is U(x) = 0.06x We set the negotiation deadline for both dentist and patient as the 10th round, and employed the NDF approach to find the agreement. In this experiment, the final agreement reached by using Fatima s approach is 13 : 20. According to the original utility functions of the dentist and the patient, such a agreement only can bring around 0.27 utility to the patient and around 0.3 utility to the dentist. Obviously, the result generated by approximating nonlinear utility function to linear utility function is worse than Ito s and our proposed approaches. Therefore, the approximating approach should be avoided when an agent s utility function has high nonlinear relationship between the issue s value and the utility Discussion In this section, we demonstrate the negotiation procedure between two agents with nonlinear utility functions by employing the proposed negotiation model. It was shown that the proposed negotiation model can efficiently handle the negotiations when agents employ the nonlinear utility functions, and successfully help agents to reach the agreement. 21

22 Round Agent Expected Maximal Counter-offers Utility Gained 1 patient : 30, 15 : 00 dentist : 00, 16 : 00 2 patient : 45, 11 : 15, 14 : 45, 15 : 15 dentist : 00, 16 : 00 3 patient : 30, 11 : 30, 14 : 30, 15 : 30 dentist : 00, 16 : 00 4 patient : 15, 11 : 45, 14 : 15, 15 : 45 dentist : 30, 16 : 30 5 patient : 00, 12 : 00, 14 : 00, 16 : 00 dentist Agreement (10 : 00 or 16 : 00) Table 2: Negotiation process between the patient and dentist agents During the experiment, two interesting phenomenons were noticed by us and worth to be discussed here. The first phenomenon is the repeating offer. It can be seen from the Table 2 that the offers 10 : 00 and 16 : 00 were exchanged between the dentist agent and the patient agent for several times before the both agent agreed on these offers. It is known that in a general negotiation, where all agents employ linear utility functions, the same offer will not be repeated by an agent. That is because if an offer is rejected by an opponent, then the same offer will definitely be rejected again by the same opponent in the following negotiation rounds. However, when agents employ the non-monotonic utility functions, the situation was changed. An agent may reject an offer in one negotiation round, but accept the same offer latter. That is because the relationship between the value of an offer and the utility gained from the offer is non-monotonic. When agents make negotiation decision based on the negotiation decision function, a rejected offer may be accepted later when agents modify their expected utility. Researches on how to eliminate or decrease the such a repeat when non-monotonic utility functions are employed may be an possible consideration to improve the efficiency of agent negotiation in nonlinear domains. The second interesting phenomenon is the multiple agreement candidates. It can be seen from Table 2 that two equivalent agreement candidates (i.e., 10 : 00 and 16 : 00) are reached finally, and agents can choose either one as the final agreement. Of course, such a problem does not exist when agents generate only one offer. However, when agents employ nonlinear utility functions, multiple equivalent offers may exchanged between agents, then how to make a reasonable selection between equivalent offers may become another interesting 22

23 research topic. Besides the utility gained from an offer, other judgements may be considered to maximise the agent s overall utility. For example, regarding to the case shown in this experiment, the dentist may choose a candidate as the final agreement by considering future possible appointments with other patients; while the patient may select between the equivalent agreement candidates by considering other possible new activities or changes on her timetable. 6. Related Work Many work on agent negotiation considering nonlinear utility have been done by other researchers. To list a few of them, Fatima et al.[12] studied bilateral multi-issue negotiation between self-interested agents whose utility functions are nonlinear. The authors argued that even though the package deal procedure leads multiple negotiation to Pareto optimal, however computing the equilibrium for the package deal procedure is not always easy, especially for nonlinear utility functions. In order to solve such a problem, the authors introduced two approaches: (i) to approximate nonlinear utility functions by linear functions, then an approximate equilibrium can easily be reached based on the approximate linear function; and (ii) to use the simultaneous procedure to negotiate issues in parallel but in dependently. By employing these two approaches, the approximate equilibrium can be found in polynomial time. This paper also showed that although the package deal procedure is known to generate Pareto optimal outcomes, the simultaneous procedure outperforms in some cases by considering economic properties. By comparing with Fatima s work, the advantages of our work is that we distinguished the differences between non-monotonic and discrete utility functions. However, we did not consider the multiple issues negotiation in this paper. Ito et al. [13] proposed a negotiation protocol for multi-issue negotiation and extended it for nonlinear utility domains. The protocol is based on a combinatorial auction protocol. In [10, 14], Ito et al. proposed an auction-based negotiation protocol among nonlinear utility agents. This model considered nonlinear utility functions, and interdependency among multi-issue. In order to reach Pareto efficiency, a sampling process is firstly performed on each negotiator s utility space, and these samples adjusted in order to get more feasible contracts. Then each negotiator makes bids, and a mediator is employed to find combinations of bids and to maximize the total value of the bids. The experimental results indicated good performance compared with other negotiation methods. By comparison with Ito s work, the advantage of our work is that we did not employ a third-part for decision making during negotiation. However, the disadvantage of our work is that we did not take the multi-issue negotiation into account. Klein et al. [15] proposed a simulated annealing based negotiation model to consider multiple inter-dependent issues and large contract spaces. In order to achieve near-optimal social welfare for negotiation with binary issue dependencies, a negotiation approach was proposed to make substantial progress towards 23

24 achieving a win-win negotiation outcome. An annealing mediator was employed to analyze negotiators feedback on their opponents offers, and to make new proposals to increase the social welfare. The difference between Klein s work and our work is that we proposed a model to bridge the gap between linear utility agents and nonlinear utility agents and a mechanism for offer generating, so as to increase the utilities for both negotiators. Robu et al. [16] proposed an agent strategy for bilateral negotiations over multi-issue with inter-dependent valuations. The authors employed ideas inspired by graph theory and probabilistic influence networks to derive efficient heuristics for multi-issue negotiations. A utility graph was employed to represent complex utility functions, and the best counter-offer was generated through searching the whole utility graph. The experimental results indicated that their proposed utility graph approach can successfully handle interdependent issues, and navigate the contract space to reach Pareto efficiency efficiently. However, their work is based on an assumption that the utility functions employed by agents are decomposable. We do not make any assumption on agent utility functions in this paper. Carbonneau et al. [17] proposed a pairwise issue modeling to predict the opponent s counter-offers during a negotiation. Issues in different types, such as continuous, discrete and ordinal, are considered. The authors employed the multiple linear regression approach to handle the counter-offers prediction when issues are continuous, and the neural network to handle the counter-offers prediction when issues are discrete and ordinal. Firstly, the negotiation issues are divided into pairs, i.e., a primary issue and a secondary issue, according to the permutations of all issues. Then for each pair, by using the historical offers and the pairwise neural network, the expected counter-offer on the primary issue in a pair can be predicted. This paper pointed out a possible solution for us when we try to extend our work to the multiple issue negotiation. Luo et al. [7] proposed a multiagent-based solution to the non-monotonic utility negotiation in meeting scheduling problem by using the Fuzzy constraints. Firstly, a coordinator agent was employed to generate a proposal time slot to all meeting participator by considering the membership function of each constraint in the scheduling and the priority of each participator. Secondly, each participator agent responded to the proposal based on their own timetable, constraints and preferences. Thirdly, the coordinator agent combined all other agents responses to a single value by using the proposed fusion operators and make the final decision on the timetable. By comparison with our approach, a coordinator agent is required in Luo s approach to take all work on proposal generation and evaluation, and decision making. As the number of participator become large, the complexity of scheduling will dramatically increase and the coordinator agent will have difficulties to generate the proposals in an efficient way. However, our approach employs decentralized decision making strategy, and each agent manage self s negotiation plan and behaviors. 24