A Trust and Reputation Framework for Game Agents: Providing a Social bias to Computer Players

Transcription

1 2016 5th Brazilian Conference on Intelligent Systems A Trust and Reputation Framework for Game Agents: Providing a Social bias to Computer Players Fábio S. do Couto Programa de Pós-Graduação em Informática Universidade Federal do Rio de Janeiro, UFRJ Rio de Janeiro, Brasil fscouto@ufrj.br Carla A. D. M. Delgado, and João C. P. da Silva Departamento de Ciência da Computação Universidade Federal do Rio de Janeiro, UFRJ Rio de Janeiro, Brasil {carla,jcps}@dcc.ufrj.br Abstract This work presents the application of trust and reputation models in the context of interactive games. This type of application aims to avoid that other players (humans or computers) can easily predict the behaviour of non-human players, and consequently loose interest in the game. In our approach, trust and reputation are mechanisms used to bring a social bias to non-human players, with the intention to emulate different types of social profiles. The main idea is to combine the social profile with the rationale the agent has regarding the game rules and game state, so that the agent uses both these traces (social profile and intelligent reasoning) in order to decide the next action to take. In this paper we present the conceptual model and the architecture of the proposed framework, and also report the results of a case study based on a game played by non-human players with different social profiles. Index Terms Autonomous Agents; Intelligent Agents; Trust; Reputation; Social Behavior; Multiplayer Games; Interactive Games I. INTRODUCTION The field of Electronic Games (EG) has come to be very important in the entertainment industry, comparable to the motion picture. Implementing games that awaken and maintain the interest of the public is a key issue for this industry. In this context, one of the most exciting areas of development is the creation of mechanisms that give to non-human players of a game, known as non player character (NPC), a rational behavior. As rational behavior we mean that a NPC should take logical decisions and interact with humans and other NPC players in a way that resembles the behavior of a human player. Providing NPCs some kind of social profile seems to be a good way to increase interest and amusement of the players of an EG. As a result, we would have NPCs that combine the usual behavior of this kind of agents with a social bias which could both make their actions less predictable and deterministic, and as well make the game more attractive. The human players would then face games where not only a rational skills or physical abilities (like shooting) are expected, but also social skills. In this work we propose a framework that implements trust and reputation mechanisms [1] as a way to provide a social bias to NPCs. It is important to note that although we had implemented our framework with some standard models, the main contribution of our work is to propose a flexible framework that can be used to implement different trust and reputation models. Trust and reputation mechanism are usually applied on e-commerce and certification chains systems, among others. In a trust and reputation system, agents build reputations of each other, based on ratings and reviews, to assess the reliability of the ones involved in a transaction. We claim that the usage of trust and reputation models in the context of EGs allows for the implementation of NPCs with the ability to use social skills in order to influence the behavior of their opponents, and also, giving the NPC s a social bias that is harder for other agents to predict. As a case study we will use the game called Settlers of Catanto analyze the proposed framework. A key point to succeed in playing Catan is trading with other players, so good interaction strategies and negotiation skills are very important. This is the reason why we chose Catan to analyze the impact of the social bias to be introduced in NPCs. This paper is organized as follows: in section II we present the trust and reputation concepts used in this work, as well as the main models and systems usually adopted. In section III we present the conceptual model and the architecture of the proposed framework. In section IV we present the simulations and result analysis. Finally, in section V, we state our conclusion and point out the possibilities for future work. II. BACKGROUND In this paper, we will use the notions of trust and reputation defined in [1], where Trust is the subjective probability by which an individual, A, expects that another individual, B, performs a given action on which its welfare depends and Reputation is what is generally said or believed about a person s or thing s character or standing. Jøsang et. al. [1] argued that the difference between these concepts can be understood in terms of the following sentences: 1) I trust you because of your good reputation ; 2) I trust you despite your bad reputation. While the first sentence illustrates the case where the decision maker trusts an agent because of its reputation, in the second sentence, despite the bad reputation of the agent, the decision maker decides to trust her. These statements can be used to define the scope of Trust and Reputation Models as follows: the Reputation Model is used to determine the /16 $ IEEE DOI /BRACIS

2 reputation of each agent, and the Trust Model, to decide whether or not to use such reputations when a decision has to be made. Both models are built from a set of interactions. A good review on trust can be found in [2], and a review on reputation and its usage appears in [3]. Reputation systems can be classified according to different criteria. Among the popular criteria is the type of architecture adopted [4], [5]. Distributed reputation systems are those where each agent has a particular view on the other agents, unlike the centralized reputation systems where there is a single repository of reputations that all agents in the system have access. In both cases, reputations are computed through direct interactions among agents and the feedback they receive. The main difference between these architectures is in the way the feedback is computed. In the centralized approach each agent considers all the reputations already computed, whereas in the distributed system each agent takes into account only the interactions in which she participated. Some of the most important trust and reputation systems and models are presented next. A. Beta Reputation System The Beta Reputation System from Jøsang et. al [6] introduced the use of uncertainty. The concept of opinion is the basis for building a reputation and is directly mapped to a probabilistic model with Beta distribution [7]. This model is recommended for scenarios with binary events. As in our framework each agent will classify its interactions with other agents as being positive or negative (thus binary), we decided to use the Beta Reputation as our basic model. B. REGRET REGRET ([8], [9]) is a formal reputation model that builds a reputation based on three dimensions, namely: individual, social and ontological. The individual dimension concerns the direct interactions that a specific agent can have with other agents. Also deals with cases where a specific agent acts as an eyewitness of the interactions that occurred among other agents in which she was not involved. The social dimension represents the reputation of a group. It is very useful when we have few or no information about the agent that is being evaluated but we have information about the group to which that agent belongs. In the ontological dimension, the idea is to combine a set of reputations that refers to different aspects, in order to form a reputation related to a more complex aspect. For instance, the reputation of an agent with respect to be a good salesman in an auction site context could be calculated as the combination of reputations of that agent with respect to the following aspects: delivery time, product price and product quality. Here we will focus only on the individual dimension which has a strong connection with our work. But we are aware that, to fully implement REGRET model in our framework, we should consider also social (group) and ontological dimensions. C. A Development Framework for Trust Models Moyano et. al ([10], [11], [12]) proposed a generic trust and reputation framework capable to support different trust and/or reputation models. This framework also dealt with the social dimension defined by [8], [9]. The framework proposed is undoubtedly powerful but complex, demands a lot of configurations, and usually generates performance gaps. As performance is crucial for a game success, this framework is not suitable to be used in the EG context. D. Pythia Pythia [13] is a reputation framework for online services. The integration of this framework in a game was not feasible for several reasons, the most relevant being the fact that Pythia was created specifically for web environments. Even for online games, Pythia has a very particular architecture based on layers and components that makes integration very hard. E. TRAVOS TRAVOS [14] is a trust and reputation model that works cohesively. It tackles the problem of lack of interactions, when there is only a few interactions among agents in the system, which results in poor quality reputation information about these agents. TRAVOS deals with this problem through an advanced model for propagating trust information. The propagation model uses recommendations to build the reputation when there is no direct information regarding the interaction among agents. Another distinguishing feature of this model is the treatment of reputations in time where it is assumed that the most recent feedback is the most valuable when building reputation. Despite our framework can give support to this feature, we decided not to incorporate it in this moment, and not to use TRAVOS for its complexity, as for our studies it was important to keep things simple, so that we could assess the effects of each aspect at a time. III. FRAMEWORK STRUCTURE In this section we will present the conceptual model, the architecture and the standard models of our framework. The framework provides Trust and Reputation Models that can be used in games in two folds: (i) exploring social behaviors of the agents (such as in The Sims [15]); (ii) as a decision support mechanism. Regarding its incorporation to already implemented NPCs, it was designed in such a way that it can be independently implemented on each NPC. Another interesting feature is that it allows easy configurations and data collection so that behavioral studies can conducted. A. Conceptual Model The central element in the conceptual model of our framework (Figure 1) is the agent or NPC. An agent is an entity that performs interactions with other entities. The conceptual model has a Trust Model and a Reputation Model. The Reputation Model uses the information about the interactions among agents to build a portfolio of reputations for each one of them. The Trust Model uses this portfolio to determine 194

3 the trust level that an agent has in others. This, together with its own reputation, will be used to influence the kind of interactions that an agent will have with other agents. Fig. 2. Framework in a distributed context Fig. 1. Conceptual Model Figure 2 illustrates the use of the framework in a game that has multiple agents (represented by the robots) and a human player. The arrows that connect the entities in the game represent the interactions between them. As an example, consider the use of our framework in the context of a game like The Sims [15] where there are several NPCs that perform actions with objects and other NPCs. In this case each NPC has its own perspective about the other players, and so different agents might attribute different reputations to each player. This characterizes the reputation system as distributed. To integrate the framework with this game, there will be an instance of the framework for each agent, with each NPC having its own reputation base. According to this schema, the feedback of each interaction an agent performs is an input for the agent s reputation model. Consider now simulation games where players are coaches of a soccer team or some other sport, where the hiring of members to teams or even the course of a game can be influenced by the reputation of each coach and/or by the performance of each player involved in a team. In this case we usually use a centralized reputation system, where all interactions with a given agent contribute to building the universal reputation of this agent. This is materialized by using a database which stores the reputations of all elements of the game (Fig. 3). So, just a single instance of the framework is used. Note that the agents are linked only through interaction arrows, having no rating arrows among them. This means that the reputation of each agent will be the same and will be available for all agents involved in the game. A centralized instance of the framework can be used at this scenario. The framework structure was designed to be extended with the creation of new Trust and/or Reputation models, as explained in the next section. Some simple models were selected to be the standard models of the framework, and will be presented later. B. Framework Architecture To present the main components of the framework, we consider a game scenario where we use a distributed approach, meaning that there will be an instance of the framework for each NPC in the game. To distinguish the agents that use our framework from those that do not, we will refer to the former as NPC and the others just as agents. The base object of the framework (see Figure 4) is called TREngine, which is the class that encapsulates the functionality of the trust and reputation models. TREngine is the communication interface between each NPC and its own framework. The TREngine main features are: store the ratings and calculate the confidence level (reputation) of each agent with whom the NPC has had interactions. When the RateAgent method is called, a new record of interactions between the NPC and the target agent is created. As a consequence, the reputation of the target agent is updated. The methods AgentReputation and AgentTrustLevel of TREngine encapsulate the functionality of the reputation and trust models and are used respectively to check the current reputation of a given agent and to use the trust model to infer the confidence level in a transaction with this agent. The trust model and the reputation model must be specified during the instantiation of a TREngine. If no models are specified, the standard models will be used, as will be described in section III-C. The engine searches for each model in the models factory to check for their availability, and then Fig. 3. Framework in a centralized context 195

4 Fig. 4. Framework Architecture instantiates them according to the choices made in the application. The factory contains all trust and reputation models implemented. If the framework customer wants to implement a new trust model (class TrustModel) or reputation model (class ReputationModel), the respective interfaces ITrustModel and IReputationModel should be used to do so, and the implemented classes must be registered in the factory through the method RegisterModel. Each time a NPC performs an action involving some other agent, the trust model uses the reputation model to check for the current target agent s reputation, and provides information about his confidence level through the AgentTrustLevel method. The Reputation model provides reputation information about the target agent through the method AgentReputation. Once an action happened, the NPC computes the result of the interaction by calling the method RateAgent. This computation is performed by gathering information about who is being evaluated and the rating that this agent received in the interaction. The rating can be a numeric value, percentage or boolean (true or false), with the latter being applied in cases where you want to use trust model directly for supporting decisions. The kind of value that will be used for the rating is defined at the time of the RateAgent method call. C. Standard Trust and Reputation Model Some models were chosen to be the framework standard models. The choices were based on simplicity and focus on models that fit well the EG context. 1) Standard Reputation Model: As mentioned before, the game scenario considered uses a distributed approach. So, each NPC should build the reputations of others based on its own information. We used the probabilistic Beta Reputation model from [6]. This model uses the probability density function (PDF) of Beta distribution to define the expected value for the next interaction with an agent, given the history of interactions with her. This is a natural choice since in the scenario we are working, each NPC should construct the reputation of its opponents according to the evolution of the game. The probability density function of the Beta distribution [16] has two parameters: α and β. The distribution function beta(α,β) can be expressed using the gamma function Γ as: Γ(α + β) f(θ α, β) = Γ(α)Γ(β) θα 1 (1 θ) β 1 where 0 θ 1,α > 0,β > 0, and with the probabilistic variable restriction p 0if α<1, and p 1if β<1. The expected value for beta distribution is E(θ) = α/(α + β). In the game context, the parameters α and β will correspond, respectively, to the quantities of positive (r) and negative (s) interactions already held with the agent being evaluated. For example, in the game for our case study, Catan, r could be the number of deals closed between a NPC and an agent i and s could be the number of deals not closed between NPC and i. Since both r and s can be 0, we have to increment their values by 1 in order to preserve E(θ): α = r +1,r 0 β = s +1,s 0 Thus it is possible to estimate agents reputations as the expected value E(θ) of the random variable beta(α,β): E(θ) =(r +1)/(r + s +2) 2) Standard Trust Model: The trust model adopted is quite simple. It comes as a model for maximizing the confidence level. The reputation obtained from the Beta reputation model may vary in the range [0, 1], and the desired confidence level (CL) for the trust model may vary in the range [ 1, 1], so we have to convert the received reputation value for the desired interval. This conversion is done by the formula: NC =(E(θ) 1 2 ) 2. When the value of the confidence level is greater than or equal to zero, the decision is to move forward with the new interaction with the target agent. The action will be denied only for cases where the trust level becomes negative. A problem arises when a NPC does not have any information about other agent. In some models, a propagation analysis takes place and so the trust information is recommended by other agents. In these cases a NPC asks for advice to other agents about a third element. However in the standard model proposed we do not use propagation models, and in the lack of information the decision is to trust the agent under analysis. The consequences of that behavior on the game is that a NPC will always trust an agent that she never met. D. Technical Issues The framework was designed always keeping in mind scalability and portability. As for scalability, despite studies have been conducted on specific models, we can observe that the basic structure allows other models to be added later. Any of the models listed in the literature review of this paper can be implemented and linked to the framework, so it is possible to make comparative studies about the performance of the application of different models in the same context. 196

5 Thinking about portability, the solution was designed as a framework mainly because there are a lot of game solutions available today that use many different types of technologies. By been flexible, the framework can be implemented in any of these technologies. IV. CASE STUDY The game Settlers of Catan 1 was chosen for our case study. In this German multiplayer board game, each player assumes the role of a settler. The goal of a player is to build and develop its own settlement. To achieve this it is necessary that each player makes trades and exchanges resources with other players. As their settlements grow, the players are rewarded and the winner is the player which reaches a specific number of points, related to the development of its settlements. As the majority of the actions a player has to take are based in trading with the other players, we chose Catan to analyze the impact of the social bias to be introduced in NPCs, as many rational implementations of NPCs for this game already exist. To analyse our framework, we use an open source Java implementation called JStellers [17], that uses AI techniques and economy models that are used in negotiation strategy planning for decision making. The strategy is maximize an object function that try to obtain more points as soon as possible, besides harm the player that has the higher score in each instance of the game. In what follows, we will call the agents originally implemented in [17] as original agents. Initially we identified the situations where the agents can interact during the game. The points of interaction identified were that an agent can (i) propose a deal to another agent (deal proposal - DP); (ii) decide to accept/reject a deal (close a deal - CD); (iii) must steal a resource from other agent (steal resource - SR) and (iv) moves the thief to a new position on the board (move thief - MT). Three social profiles were then designed: trust-reputation, cooperative and anti-social agent profiles. The trust-reputation (TR) agents (agents that instantiate the TR profile) use the Reputation and Trust Models presented before in the decision making process. Decisions taken by each agent at the points of interaction serve as inputs for the construction of reputations by TR agents. The Trust Model is used when a TR agent has to take a decision at the points of interaction. So, a TR agent acts in twofold: (i) collecting information about the action of its opponents (constructing a reputation of them) and (ii) using this reputation on the Trust Model to define its own course of action. In the reputation construction phase, a TR agent i will consider a positive interaction (increment r by 1) with an agent j when (i) j moves the thief for a place where i has no settlements, (ii) j does not steal resources from i or (iii) j accepts to close a deal with i. Otherwise, s consider a negative interaction (increment s by 1) with i. The Trust Model is consulted when: (i) an agent j asks to a TR agent i to close 1 The complete game description and rules can be found at the website a deal and i has to decide whether (positive trust on j) or not (negative trust on j) to close, (ii) to send a deal proposal only to agents with a positive level of trust, (iii) to harm agents with a high negative level of trust. The cooperative profile has a friendly, altruist behavior. An agent that implements this profile always choose the actions that has the higher potential of help other agents or, if she can not help, she tries to cause the least damage. For example, in situations that she has to move the thief, she chooses places where there are no settlements. The only situation that this kind of agent can harm someone is when she has (with no option not to do so) to steal resources from someone, and in that case we set the choice of whom she will steel from randomly. Agents that implement the Anti-social profile act in the opposite way to cooperative agents. Their decisions seek to harm as many agents as possible and they avoid any negotiation unless those initiated by themselves. She relocates the thief to a place that causes more damage to most of other agents and when she has to harm an opponent, she chooses the one who is best placed in the game as the original agents in JSettlers do. The way each of the three social profiles act in points of interaction are summarized in Table I. DP CD SR MT TABLE I BEHAVIOR OF AGENTS IN POINTS OF INTERACTION OF THE GAME Cooperative Anti-social Trust-Reputation proposes to proposes only when proposes to agents all agents needs a resource she trusts Always Always accepts - pos. trust accepts rejects reject - neg. trust steals from agent steals from agent steals from agent that has high score with higher score that she trusts less less more settlements move close to agent settlements of others that she trusts less In order to evaluate the performance of the agents, we identify three aspect of the game that should be considered: (i) location - advantage of a position on the board, since some places could increase the chances of obtaining resources. This is evaluated by the amount of resources an agent acquires along a match; (ii) negotiation factor that measures when an agent obtains resources from negotiation or as a consequence of rolling of dice. This is evaluated considering the percentage of the amount of resources acquiring from negotiation wrt all acquired resources by an agent; (iii) bargaining power, that measures the agent s ability (social bias) to close deals. In this case, we considered the number of proposals that an agent (a) makes to others, (b) receives from others and (c) the number of deals she closed. This quantity evaluation was used as basis for a quality analysis of the behavior of the agents, that we will present in what follows. Initially, we run a simulation with all parameters fixed (for example, the sequence of results of rolling dice, the position of the agents on board and the order of play) and considering only 4 players, all of them original (jsetlers ) agents. In this scenario, the agent who wins the match concentrates the majority (83%) of deals closed (all of them that she proposed, 197

6 not closed any deal offered by others), obtaining about 25% of its resources through negotiation. The second step was to introduce different kinds of social agents in the game to evaluate how was the interactions with original agents. Starting with TR agents, we observe that normally TR agents classify other TR agents as reliable and when a TR agent was classified as neutral by other TR, the reciprocal also happened. An interest situation occurs that favors original agents when a TR agent i has a positive reputation about an original agent j. In this case, the reputation of i from the point of view of another TR agent k decreases, since k seems to consider the alliance between i and j as negative for its interests, which has as consequence the enhancing of the alliance between them. We also observed that there was a great swing regarding reputation that TR agents had about the original agents, and as more TR agents participate in the match, the alliances among them tend to be more unstable. In matches where we have only original and cooperative agents, the latter tend to form clusters with great number of deals closed among them and, as consequence, always a cooperative agent wins. When we consider only original and anti-social agents, the number of wins is divided fifty to fifty. In situations where we have all kinds of agents, the cooperatives tend to have a better performance, since TR agents tend to help them, followed by TR, original and antisocial agents. We also considered a dynamic environment (without fixing the game parameters), as in a normal match. We did a total of 2360 matches that involved original agents and a combination of one (1110 matches), two (1050 matches) or three (200 matches) types of the other agents. We observed an impressive improvement of the performance of anti-social agents, placing then only below the cooperatives. This is explained by the fact that the RP agents are very tolerant to anti-socials. This could be different if we had given more weight to closed deals in the Trust-Reputation Model. The anti-social strategy affects the cooperatives only when the former outnumber the latter. The TR strategy works well against original agents since it can prevent original agents to harm TR agents. V. CONCLUSION AND FUTURE WORKS We presented in this work a framework for using trust and reputation to model and implement social profiles in NPCs. The TR framework we proposed was instantiated by a flexible, configurable and scalable architecture, facilitating its use by game developers. As a flexible framework, any model, like the aforementioned or other models of trust and reputation can be implemented and linked to our framework. Our framework was used to conduct a case study with the Catan game. We got an available implementation for an NPC for Catan and instantiated three different social profiles using our framework. As our case study showed, the social agents can bring unpredictability to the game evolution, as the result of the game can be influenced by the combination of the social profiles of the players, and also, by the sequence of social interactions that happens in a match. The more points of interaction a game has, and the greater the impact of interactions for the game result, the greater will be the impact of the social bias introduced in the NPCs. This confirms our initial hypothesis: the introduction of a social bias in NPCs can bring unpredictability to the EG and allow for introducing social skills as another skill a player might use to succeed in the game. As future work, we plan to run experiments bringing together human and non-human players, so that we can assess if the social bias does indeed make the game more interesting to human players. Also, we plan to experiment our framework with other games, for example poker and other card games. Running experiments with games like The Sims is not feasible by now, as no open interface to the game agents is provided. It is also in our plans to analyze our results using a game theoretic framework. REFERENCES [1] A. Jøsang, R. Ismail, and C. Boyd, A survey of trust and reputation systems for online service provision, Decision Support Systems, Mar [2] C. Castelfranchi and R. Falcone, Trust Theory: A Socio-Cognitive and Computational Model, 1st ed. Wiley Publishing, [3] R. Conte and M. Paolucci, Reputation in Artificial Societies: Social Beliefs for Social Order, ser. Multiagent Systems, Artificial Societies, and Simulated Organizations. Springer US, [Online]. Available: [4] L. Mui, M. Mohtashemi, C. Ang, P. Szolovits, and A. Halberstadt, Ratings in distributed systems: A bayesian approach, in Proceedings of the Workshop on Information Technologies and Systems (WITS), [5] L. Mui, M. Mohtashemi, and A. Halberstadt, Notions of reputation in multi-agent systems: a review, in Proceedings of the First Int. Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS), Jul [6] A. Jøsang and R. Ismail, The beta reputation system, in Proceedings of the 15th Bled Electronic Commerce Conference, Jun [7] G. Casella and R. L. Berger, Statistical Inference. Duxbury Press, [8] J. Sabater and C. Sierra, Regret: A reputation model for gregarious societies, in Proceedings of the 4th Int. Workshop on Deception, Fraud and Trust in Agent Societies, in the 5th Int. Conference on Autonomous Agents (AGENTS 01). Montreal, Canada: ACM Press, [9], Reputation and social network analysis in multi-agent systems, in Proceedings of the First Int. Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS), Jul [10] F. Moyano, C. Fernandez-Gago, and J. Lopez, Building trust and reputation in: A development framework for trust models implementation, in 8th International Workshop on Security and Trust Management (STM 2012), Pisa, In Press. [11], A conceptual framework for trust models, in 9th International Conference on Trust, Privacy & Security in Digital Business (TrustBus 2012), ser. Lectures Notes in Computer Science, S. Fischer-Hübner, S. Katsikas, and G. Quirchmayr, Eds., vol. 7449, Springer Verlag. Vienna: Springer Verlag, Sep , pp [12], Implementing trust and reputation systems: A framework for developers usage, in International Workshop on Quantitative Aspects in Security Assurance, Pisa, Sep [13] P. J. Windley, T. Kevin, and D. Daley, A framework for building reputation systems, in Www2007, Banff, Canada, [14] W. Teacy, J. Patel, N. Jennings, and M. Luck, Travos: Trust and reputation in the context of inaccurate information sources, Autonomous Agents and Multi-Agent Systems, vol. 12, [15] (2012) The sims game portal. Electronic Arts. [Online]. Available: US/home [16] A. Jøsang, A logic for uncertain probabilities, in International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, Jun [17] R. S. Thomas, Real-time decision making for adversarial environments using a plan-based heuristic, Ph.D. dissertation, Evanston, IL, USA,