Networks of Influence Diagrams: A Formalism for Representing Agents Beliefs and Decision-Making Processes

Transcription

1 Networks of Influence Diagrams: A Formalism for Representing Agents Beliefs and Decision-Making Processes By Ya akov Gal and Avi Pfeffer in JAIR (2008), AAAI08 Tutorial Presented by Chenghui Cai Duke University, ECE December 8, 2008

2 Outline Introduction Single-agent decision-making decision theory influence diagrams Multi-agent decision-making multi-agent influence diagrams networks of influence diagrams NID application: RoShamBo NID Relationship with economic models Conclusions

3 Introduction Goals: computer agents to make good decisions when interacting with environments other computer agents people networks of people and computers Challenges large and uncertain environments numerous and complex decisions other decision makers (e.g., agents and people)

4 Graphical Models Graphical Models Can meet the challenges! natural and compact representation of decision-making under uncertainty decompose complex decision-making problems support recursion and divide-and-conquer techniques Themes representation: creating a probabilistic model of agents decision-making processes inference: computing strategies for agents

5 decision theory Decision theory Basis: uncertainty (probability) + utility Example 1 Bob observes the tomorrow s weather forecast from an expert before deciding whether to carry an umbrella to work tomorrow. Bob wishes to stay dry, but carrying an umbrella around is annoying. Forecast

6 decision theory Example 1 Set A of actions: Umbrella UM = {y, n} Set S of unobserved events: Weather W = {sun, rain} Set O of observations: Forecast F = {sun, rain} Probability distribution over: events P(s); observations given events P(o s) Utility function U maps from actions and events,s A to real numbers R W = sun W = rain W F = sun F = rain sun rain W UM U sun y -10 sun n 100 rain y 100 rain n -10

7 decision theory Choosing the Best Action a LetU a (Bob s) be Bob s reward for taking action a A after event s S has occurred. The expected utility for Bob after observing o O is: EU a (Bob o) = s S P(s o) U a (Bob s) (1), where P(s o) = P(s)P(o s)/p(o). Optimal behavior Given observation o, choose the best action a that leads to the maximum expected utility a = argmax EU a (Bob o) (2) a A

8 decision theory Computing an Optimal Strategy for Bob A strategy for Bob must specify whether to take an umbrella for any possible value of the forecast. Suppose that F = sun, Marginal probability P(F = sun) = P(F = sun W = sun)p(w = sun) + P(F = sun W = rain)p(w = rain) = = 0.58 Bayes rule P(W = sun F = sun) = P(F = sun W = sun)p(w = sun)/p(f = sun) = 0.72 P(W = rain F = sun) = 0.28

9 decision theory Computing an Optimal Strategy for Bob Suppose that F = sun, P(F = sun) = 0.58, P(W = sun F = sun) = 0.72, P(W = rain F = sun) = 0.28 EU UM=y (Bob F = sun) = P(W = sun F = sun) U UM=y (Bob W = sun) + P(W = rain F = sun) U UM=y (Bob W = rain) = 0.72 ( 10) = 20.8 EU UM=n (Bob F = sun) = P(W = sun F = sun) U UM=n (Bob W = sun) + P(W = rain F = sun) U UM=n (Bob W = rain) = 0.72 (100) ( 10) = 69.2 If F = sun, EU UM=n (Bob F = sun) > EU UM=y (Bob F = sun),then UM = n for Bob

10 decision theory Computing an Optimal Strategy for Bob Suppose that F = rain, P(F = rain) = 0.42, P(W = sun F = rain) = 0.43, P(W = rain F = rain) = 0.57 EU UM=y (Bob F = rain) = 52.7 EU UM=n (Bob F = rain) = 37.3 If F = rain, EU UM=y (Bob F = rain) > EU UM=n (Bob F = rain), then UM = y for Bob Strategy for Bob: F = sun F = rain UM n y

11 decision theory Making Sequential Decisions, Extended Example 1 The newspaper forecast is more reliable, but costs money, decreasing Bob s utility by 10 units. Now two decisions: NP = {y, n} UM = {y, n} Choosing the best action for one decision depends on the action for the other decision. How to weigh the tradeoff between these two decisions? W F = sun F = rain sun rain W NP UM U sun y y -20 sun y n 90 rain y y 90 rain y n

12 decision theory When NP = y, Marginal probability P NP=y (F = sun) = P NP=y (F = sun W = sun) P(W = sun) + P NP=y (F = sun W = rain) P(W = rain) = = 0.56 Bayes rule P NP=y (W = sun F = sun) = P NP=y (F = sun W = sun) P(W = sun)/p NP=y (F = sun) = 0.86 P NP=y (W = rain F = sun) = 0.14 Expected Utility EU NP=y,UM=y (Bob F = sun) = P NP=y (W = sun F = sun) U NP=y,UM=y (Bob W = sun) + P(W = rain F = sun) U NP=y,UM=y (Bob W = rain) = 0.86 ( 20) = 6.4 EU NP=y,UM=n (Bob F = sun) = 74.6

13 Decision Tree Decision Tree y NP n F PPNP=y (F=sun) sun rain sun F 0.42 rain UM UM UM UM y n y n y n y n EU NP=y, UM=y (Bob F=sun)

14 Decision Tree Solving Decision Tree Backward Induction 0.56* *60.3 = 68.3 y NP n 0.58* *52.7 = 62.3 F F PPNP=y (F=sun) sun sun rain rain UM UM UM UM y n y n y n y n EU NP=y, UM=y (Bob F=sun)

15 Influence diagrams (ID) ID ID: compact graphical and mathematical representation of a decision situation; probabilistic inference + decision making; maximize expected utility Rectangles are decisions; ovals are chance variables; diamonds are utility functions Each chance node specifies a probability distribution (CPD) given each value of parents F UM W U

16 Influence diagrams (ID) ID Parents of decisions (informational parents) represent observations Parents of chance nodes represent probabilistic dependence Parents of utility nodes represent the parameters of the utility functions A strategy for a decision is a function from its informational parents to a choice for the decision. For each observation, a pure strategy prescribes a single choice of action for an agent F UM W U

17 Influence diagrams (ID) Influence Diagram for Example 1, Umbrella Scenario F UM W U W = sun W = rain W F = sun F = rain sun rain W UM U sun y -10 sun n 100 rain y 100 rain n -10

18 Influence diagrams (ID) Influence Diagram for Extended Umbrella Scenario No forgetting edges added from NP to UM Agents remember their past decisions when they make future decisions Information available to past decisions is also available to future decisions NP F UM W U

19 Influence diagrams (ID) Converting ID to Decision Tree: Extended Umbrella Example NP y n F F PPNP=y (F=sun) sun rain sun 0.42 rain NP F W UM UM UM UM y n y n y n y n EU NP=y, UM=y (Bob F=sun) Disadvantage : Lose the graph structure UM U

20 multi-agent influence diagrams Example 2 Proposer can offer some split of 3 coins to Responder. If Responder accepts, offer is enforced; if Responder rejects, both receive nothing. Offer may be corrupted and set to (1,2) split (proposer/responder) by noisy channel with 0.1 probability.

21 multi-agent influence diagrams MAID [Milch and Koller Multi%agent IJCAI01] In'uence Diagrams Extend Influence Diagrams to the multi-agent case Rectangles and diamonds represent decisions and utilities associated with agents, respectively; ovals represent chance variables Responder Extend In'uence Diagrams to the multi%agent case. Rectangles and diamonds represent decisions and utilities associated with agents; ovals represent chance variables. A strategy for a decision is a mapping from the informational parents of the A strategy for a decision decision isto aa mapping value its from the informational domain. parents of the decision to a value in its domain A strategy pro&le includes strategies for all decisions. A strategy profile includes strategies for 100 all decisions Proposer U(Responder) channel U(Proposer) Channel Responder Proposer!0,3"!1,2"!2,1"!3,0"!0,3" !1,2" !2,1" !3,0"

22 multi-agent influence diagrams Solving MAID by Converting MAID to Decision Tree Solve Response and determine strategy for Response: accept any split larger than zero Solve Proposer and Offer is the largest split for proposer that offers a positive share to responder 1

23 networks of influence diagrams Traditional Game Theory Limitations Game Theory Assumptions rational Common knowledge of game structure Agents beliefs correct/consistent Real World Agents maybe irrational Uncertain about game, other s strategies Agents belief might incorrect

24 networks of influence diagrams What we need Language for representing uncertainty over decision making must allow for distinction between agents models of each other and how they actually behave incorrect/inconsistent beliefs; using heuristics representation of belief hierarchies, e.g., I believe that you believe that... framework for learning

25 networks of influence diagrams To motivate single-agent NID, consider Example 3 Bob observes the forecast before deciding whether to take an umbrella when leaving the house. In reality, forecasters are quite trustworthy. We wish to model the fact that Bob is less trusting of forecaster than he should be. What is Bobs strategy given his wrong belief about forecasters?

26 networks of influence diagrams Bob s Utility Utility Top-level block Bob s block Mod[UM] = Bob sblock, means Bob may be using Bob s block to compute strategy to make decision UM Edge represents Bob (agent) at Top-Level block (source block) modeling decision UM as being made in Bob s block (target block)

27 networks of influence diagrams A NID is a directed, possibly cyclic graph, in which each node is a MAID. Call the nodes of a NID blocks. They are different mental models. A mental model for an agent may itself use descriptions of the mental models of other agents. Let D be a decision belonging to agent α in block K, and let β be any agent. (In particular, β may be agent α itself.) Utility Top-level block Bob s Utility Bob s block

28 networks of influence diagrams Introduce a new type of node, denoted by Mod[β, D] with values that range over each block L in NID. When Mod[β, D] = L, β believes that α may be using the strategy computed in block L to make decision D A Mod node is a chance node just like any other; it may influence, or be influenced by other nodes of K Solving NID by converting to MAID Utility Top-level block Bob s Utility Bob s block

29 Table 10: Payoff Matrix for Rock-paper-scissors Opponent Modeling Bob,STEAL Top level Alice,PITCHOUT Rock Paper Scissors Competition: Multi-agent Case In opponent modeling, agents L try to learn the patterns exhibited by other players and react to their model of others and thus do better. Example 4 In the(c) game Cyclic of RoShamBo NID (commonly referred to as Rock-Paper-Scissors), Alice and Bob simultaneously choose one of Figure rock, 9: paper, Cyclic or Baseball scissors. If Scenario they choose (Example the same item, 4.5) the result is a tie; otherwise rock crushes scissors, paper covers rock, or scissors cut paper, as shown in the table rock paper scissors rock (0, 0) ( 1, 1) (1, 1) paper (1, 1) (0, 0) ( 1, 1) scissors ( 1, 1) (1, 1) (0, 0)

30 Opponent Modeling Bob s reasoning Alice and Bob are playing rounds of rock-paper-scissors. Suppose there exists a signal S that depends on prior history. Strategy for Bob BR(S) =scissors BR(BR(BR(S)))=paper BR(..(S)..))=rock Strategy for Alice S = paper (e.g.) BR(BR(S))=rock BR(BR(BR(BR(S))))=scissors Modeling double guess, triple guess, like I think that you think...

31 Opponent Modeling NID Bob modeling Alice Alice Automaton K1 rock B paper Nodes in NIDs are called blocks. Each block represent a separate decision-making process An edge represents an agent at the source block modeling a decision as being made in the target block. The edge leads from the modeled decision to the target block and is labeled with the modeling agent 1 1

32 Opponent Modeling Alice s double guessing Bob Bob modeling Alice K1 Alice Automaton B K2 A Alice modeling Bob scissors 1

33 Opponent Modeling Bob s double guessing Alice Bob modeling Alice K1 Alice Automaton B A Bob modeling Alice K2 Alice modeling Bob rock B 1

34 Opponent Modeling Alice s triple guessing Bob K3 Alice modeling Bob paper K1 Alice Automaton rock B Bob modeling Alice paper A A Bob modeling Alice K2 Alice modeling Bob rock B scissors

35 Opponent Modeling RoShamBo NID K3 A Alice modeling Bob paper B TL K1 Alice Automaton B rock B Bob modeling Alice paper A Bob modeling Alice K2 Alice modeling Bob rock B B scissors

36 Opponent Modeling Empirical Evaluation Pick nine top contestants from the first automatic RoShamBo Networks of Influence Diagrams Competition; 3000 rounds with each contestant; +1 for winning a round, -1 for losing one) Average Score Difference Contestant Opponent type Number Iocaine Powder 1 By Ya akov Probabilistic, Gal and Avi Pfeffer Pattern, in JAIR Exploitative (2008), AAAI08 2, 9Tutorial Presented by Chenghui Cai Duke University, ECE: Networks of Deterministic, Influence Diagrams: Pattern, AExploitative Formalism for Representing 3, 6, 5 Agents Beliefs and Decision-Making Processes

37 Conclusions - 1 Conclusions: Building blocks of NIDs are MAIDs In NIDs, each mental model itself is a graphical model of a game. Agent in one mental model may believe that another agent (or possibly itself) uses a different mental model to make decisions Relationship between NID and Bayesian games: they are equally expressive, but NIDs may be exponentially more compact NIDs can describe agents who play irrationally, represent players inconsistent and/or incorrect beliefs I believe that you believe type reasoning

38 Conclusions - 2 Conclusions: NID can be used to learn non-stationary strategies in rock-paper-scissors Models inspired by NIDs can learn people s play in negotiation games Focus of our continuing work will be to develop a general method for learning models in NIDs Chenghui s Remark: something like Dynamic NID to represent multiagent sequential decision process or multiagent POMDP?

39 Conclusions - 2 NID Converted to MAID Conclusions: Any NID can be converted to a MAID But MAID is hard to construct directly