DISTRIBUTED ARTIFICIAL INTELLIGENCE

Transcription

1 DISTRIBUTED ARTIFICIAL INTELLIGENCE LECTURE 3: PROBLEM MODELING INTRODUCTION There are different approaches for modeling of agent-based systems. The model that has gained most attention ti is modeling agents as Utility Maximizers who inhabit some kind of Markov decision process. By: Adel Akbarimajd University of Mohaghegh Ardabili The reason is probably bl due to its flexibility as well as its well established roots in game theory. 2 UTILITY FUNCTION PREFERENCE ORDERING An agent s preferences are captured by a utility function. Utility ty function: A map from the states of the world to a real number. Agent can define a preference ordering over the states of the world. This ordering has the following properties: DA AI-:03-Problem Modeling The states t are defined d as those states t of the world that the agent can perceive. 3 Alltheagentshavetodoistakeactions which maximize their utility. 4

2 TRANSITION FUNCTION POLICY Assume that the agent does know the probability of reaching state s' given that it is in state s and takes action a. This probability is given by T(s, a, s') which we call the transition function. The transition function returns a probability then the sumof T(s, a, s') over all possible a and s' is equal to one. Expected utility for taking action a in state s is defined as: 5 6 DAI-:03-Problem Modeling VALUE OF INFORMATION VALUE OF INFORMATION An agent can use the expected utility function to determine the value of a piece of added d information it might acquire. Consider a piece of new information that tells the agent is not really in state s but it is instead in state t. Value of information provides a simple and robust way to make meta-level decisions: What knowledge to seek. What messages to send Which h sensors to turn on How much deliberation to perform etc. The value of this information is given by: 7 An agent can use it to play what if games to determine its best action Example: Doctor-test-prescription 8

3 MARKOV DECISION PROCESSES MARKOV DECISION PROCESSES A simplifying assumption: the choice of the new state depends only on the: Agent's current state The agent's action. Definition (Markov Decision Process): An MDP consists of: An initial state s 1 taken from a set of states S A transition function T(s, a, s') A example of graphical representation: DAI-:03-Problem Modeling A reward function r : S R 9 10 DISCOUNTED REWARDS OPTIMAL POLICY How to handle future rewards? Reduce the impact of rewards that are aefarther of in the future. Multiplying the agent's future rewards by discount factor represented by Note that the agent only knows s 1. The rest of the states, depend on the transition function T. So we can rewrite optimal policy as: DA AI-:03-Problem Modeling If an agent starts from state s 1 and visits states s 1, s 2, s 3, then its discounted reward is given by where u(s') is the utility the agent can expect from reaching s' and then continuing on to get more rewards for successive states while using *

4 BELLMAN EQUATION Utility of state s is composed of two terms: Immediate Reward: When the agent arrives in s it receives a reward of r(s). Future discounted rewards: Because it is in s it can take its action based on *(s) ) and will get a new reward at the next time (Utility of being in state s) As such, we can define the real utility the agent receives for being in state s as: 13 VALUE ITERATION To find values of u(s): Given n states we have n Bellman equations each one with a different variable. We thus have a system of n equations with n variables. Theoretically, e we can find values for all these variables, a however, solving this set of equations is not easy. Value iteration: We start by setting the values of u(s) to arbitrary numbers. Then iteratively improve these numbers using: It has been shown that this process will eventually, and 14 often rapidly, converge to the real values of u(s). VALUE ITERATION Example: 15 DAI-:03-Problem Modeling PLANNING In the planning problem an agent is also given a set of possible states and is asked for a sequence of actions that will lead it to a desirable world state. However, instead of a transition function the agent is given a set of operators. Each operator has pre-requisites requisites (that specify when it can be used in which states) and effects (which specify the changes the operator will cause in the state). The planning problem is to find a sequence of operators that take the agent from the start state to the goal state. It should be clear that this problem is a special case of an MDP, one where only one state provides a reward and all the transitions have probability of 1 or 0. 16

5 MULTIAGENT MDPS Dependent agents: Other agents are not considered as environment for a given agent: Instead of having a transition function T(s, a, s') we have a transition function T(s, a, s'). a is joint action, a vector of size of the number of agents where each element is an agent's action. We need to determine how the reward r(s) is to be doled out amongst the agents. 1. A possible way is to divide it evenly among the agents. 2. A better method is to give each agent a reward proportional to his contribution tibti to the system's reward. 17 Agent does not know in which state it is in but, instead, believes that it can be in any number of states with certain probability. We can capture this problem by modeling the agent's belief state instead b of the world state s. This belief state is merely a probability distribution over the set of possible states and it indicates the agent's belief that it is in that state. For the case with four states, the vector b =[0.5,0.5,0,0] indicates that the agent believes it is either in s 1 or s 2, with equal probability, and 18 that it is definitely i not in s 3 or s 4. We also define an observation model O(s; o) which tells the agent the probability that it will perceive observation o when in state s. Specially, if the agent's current belief is b and it takes action a then its new belief vector b can be determined using: where b ( s ) is the value of b for s and is a normalizing constant that makes the belief state sum to 1. When we put all these requirements together we have 19 a partially observable Markov decision process 20

6

7 25 26 MULTIAGENT MDPS Dependent agents: Other agents are not considered as environment for a given agent: Instead of having a transition function T(s, a, s') we have a transition function T(s, a, s'). a is joint action, a vector of size of the number of agents where each element is an agent's action. We need to determine how the reward r(s) is to be doled out amongst the agents. 1. A possible way is to divide it evenly among the agents. 2. A better method is to give each agent a reward proportional to his contribution tibti to the system's reward. 27