Monte Carlo AI for Air Traffic. Kagan Tumer MCAI 2013 March 20, 2013

Transcription

1 Monte Carlo AI for Air Traffic Kagan Tumer MCAI 2013 March 20, 2013

2 Air Traffic Conges:on

3 Traffic: Interesting Problem? Problem inherently large and distributed Many cars Many intersections Many planes Good decisions change with time Daily scale Yearly scale Solution needs to be adaptive Sensor lights aren t very bright Solution needs to scale: One intersection A hundred intersections Huge amount of uncertainty Weather Events

4 Language Agent: an algorithm that will sense, decide, act on the environment Multiagent systems: Many agents interact Interactions create unpredictable learning environment Air traffic is a great example of a large, stochastic multiagent systems

5 Intelligent Agents in Traffic Highway traffic Drivers Lanes City Traffic Lights Cars (not drivers) Air Traffic Gates Routes Aircraft Role of air traffic controllers

6 Air Traffic Flow Management Current Situation 40,000+ flights operate in the US airspace in one day Delays caused by weather and airport conditions: 1,682,700 hours of delay (2007) 740,000,000 gallons of fuel wasted (2007) Estimated cost impact: over $41 billion (2007) Moving forward Threefold increase in air traffic Increased heterogeneity of aircraft Need Algorithmic solution Infrastructure will not change significantly

7 Current Air Traffic Management Air Traffic decisions made at four levels: 1. Airspace Management (6 hours to 1 year) Game Plan Centralized 2. National Flow (2-8 hours) Centralized 3. Regional Flow (20 min-2 hours) Hierarchical 4. Separation Assurance (2-30 minutes) Air traffic controllers

8 Current Air Traffic Management Air Traffic decisions made at four levels: 1. Airspace Management (6 hours to 1 year) Game Plan Centralized 2. National Flow (2-8 hours) Centralized 3. Regional Flow (20 min-2 hours) Hierarchical 4. Separation Assurance (2-30 minutes) Air traffic controllers

9 Multiagent Learning for Air Traffic? Advantages: Large distributed problem Naturally decentralized Human senses are overwhelmed by data Challenges: Humans have to remain in the loop Agent approach needs to be transparent Allow humans to take over Help humans don t replace them

10 Snapshot of the airspace

11 First steps What are we measuring? System performance? (reward/objective/utility/evaluation) How are we measuring it? System snapshots (state) System movie What about system dynamics? Simulators

12 What are we after? How do we know if we succeed? Define a system level objective Minimize congestion What about delays?

13 System Objective Function Minimize congestion C(z) = C s (z) s S C s (z) = t (k s,t c s ) 2 I ks,t >c s Minimize delays B(z) = B (z) a a A B a (z) = (t a τ a ) I ta >τ a

14 System Objective Function Full state vector Lateness-congestion tradeoff coefficient Lateness Term Congestion Term

15 Simulation: FACET

16 FEATS: Fast Event-based Air Traffic Simulator Simulation of generic or US airspace Developed at Oregon State University Can simulate up to 28,000 flights per second Designed for Monte Carlo simulations and learning Learning may occur: After a complete simulation At discrete times during the simulation (use the LEARN event) By triggering some other event Simulation: FEATS

17 We need 4 more things Learning Multiagent Approach

18 Agent-Based Air Traffic Management 1. Identify agents 2. Identify actions 3. Derive agent objective functions 4. Select agent learning algorithm

19 Approach: Identify Agents Agents as aircraft? agents Little data to train agents Actions conflict with pilots Agents as routes? Not well defined agents Actions of routes? Agents as fix locations? Number of agents vary with need All flight plans contain at least one agent fix. Agents have simple actions: set metering restrictions Agents can be active or inactive (e.g., live around congestion).

20 Approach: Identify Agents Agent Fix min MIT min MIT min MIT Agents as fix locations? Number of agents vary with need All flight plans contain at least one agent fix. Agents have simple actions: set metering restrictions Agents can be active or inactive (e.g., live around congestion).

21 Agent-Based Air Traffic Management 1. Identify agents Fixes 2. Identify actions Miles in Trail Ground holds Reroutes 3. Derive agent objective functions Difference objective 4. Select agent learning algorithm Simple reinforcement learning

22 Basic Algorithm An agent keeps table of Values for each ac:on: V(a) Policy: With probability epsilon choose random ac:on Otherwise choose ac:on with highest value Agent takes an ac:on and receives a reward R Value update: V(a) (1 α) V(a) + α R

23 Difference Reward Look at difference between system reward, and system reward with agent taking constant action c i D i ( z) = G( z) G( z + c ) i i System Reward System Reward Without i s influence D is hard to compute: D requires n + 1 runs of FACET for every learning episode G requires 1 run Solution: Estimate difference reward

24 Difference Reward Look at difference between system reward, and system reward with agent taking constant action c i D i ( z) = G( z) G( z + c ) i i Key theoretical result: G(z i + c i ) z i = 0 g i (z) z i = G(z) z i

25 Difference Reward Look at difference between system reward, and system reward with agent taking constant action c i D i ( z) = G( z) G( z + c ) i i Key theoretical result: D and G are aligned: What s good for me is good for the system

26 Experiments & Algorithms Experiments: Two Artificially Created Congestions One heavy congestion One light congestion 300 aircraft Algorithms: Monte Carlo Estimation Q-learning agents using: G - system reward D - difference reward (local) Dest1 - first estimate of difference reward Dest2 - second estimate of difference reward

27 Results: 20 Agents

28 Penalty Tradeoff Delay Only Congestion Only

29 Scaling Maximum System Utility Achieved D D est G Monte Carlo Number of Agents

30 More advanced results What if there are humans in the loop? Suggestion agents What if you can t compute the objective functions? Model system objectives

31 Agent Decision Making Input Agent Ac:on Reward Input Agent Intended Ac:on Noise Actual Ac:on Actual Reward Reward Received by Agent Noise

32 Human in the Loop: Suggestion Agents Input Agent Intended Ac:on Noise Agent Ac:on Filter Actual Ac:on Actual Reward Reward Received by Agent Noise

33 Suggestion Agents Maximum System Reward Achieved Agent Solution Minimizing Congestion and Lateness Agents Give Suggestions (using D) Agents Give Suggestions (using G) Human Models Minimizing Congestion Only Number of Steps

34 Suggestion Agents Final Maximum System Reward Achieved Agents Give Suggestions (using D) Agents Give Suggestions (using G) Agents Only Human Models Only Weight of Agents Suggestions

35 What if you can t compute the objectives? Use a function approximation Tabular linear function Non-linear approximator (like a neural network)

36 Modeling Objectives

37 Modeling Objectives

38 Summary of Air Traffic Project Lessons Learned: Key of successful application is in selecting: Agents Obvious choice is not always the best Agent actions Modify system parameters Agent rewards Good choice has direct impact on performance Don t rock the boat too much: Humans have to remain in the loop Agent approach needs to be transparent Allow humans to take over

39 Acknowledgements and Publica:ons Collabora:ve work spanning over 7 years NASA Ames Research Center, UARC Adrian Agogino Oregon State University ScoQ Proper, A:l Iscen, Carrie Rebhuhn, Will Curran, Zach Welch Select papers: A Mul:agent Approach to Managing Air Traffic Flow. Journal of Autonomous Agents and Mul2agent Systems, 24:1-25, 2012 Learning Indirect Ac:ons in Complex Domains: Ac:on Sugges:ons for Air Traffic Control. Advances in Complex Systems, 12: , Regula:ng Air Traffic Flow with Coupled Agents. In Proceedings of the Seventh Interna2onal Joint Conference on Autonomous Agents and Mul2agent Systems. Estoril, Portugal, May Distributed Agent- Based Air Traffic Flow Management. In Proceedings of the Sixth Interna2onal Joint Conference on Autonomous Agents and Mul2agent Systems, pages Honolulu, HI, May NSF support: Grants CNS and CNS- II- EN

40 Ques:ons? Kagan Tumer hqp://web.engr.oregonstate.edu/~ktumer/ AADI: Autonomous Agents and Distributed Intelligence Lab