World Modelers. Paul R. Cohen Information Innovation Office, DARPA

Transcription

1 World Modelers Paul R. Cohen Information Innovation Office, DARPA

2 The Goal of World Modelers World Modelers technology will enable analysts to build models rapidly to analyze questions relevant to national and global. World Modelers analyses will be comprehensive, causal, probabilistic, and timely enough to recommend specific actions that could avert crises. Approved for Public Release, Distribution A 2

3 A Food Shortage Forecast Example Task: For any subnational location (e.g., Southern Sudan) generate food shortage scenarios two years out. Consider a comprehensive set of causes. Integrate current and historical data and human expert analysis. Model the scenarios quantitatively and probabilistically. Make forecasts, explain sources of uncertainty, and identify and explain specific potential solutions. Update models, forecasts and probabilities on receipt of new data. Large organizations have spent years perfecting analytical methods that do some of the above. World Modelers technology is expected to build models to solve this problem and many others like it in a month. What/Who Factors Considered Scale Methodology Horizon Time to Prepare FEWS-NET 1 / USAID Comprehensive Subnational Qualitative 3-6mo. 3-6mo.? GIEWS 2 / FAO U.N. Comprehensive Subnational Qualitative 3-6mo. 3-6mo.? Reports / OECD-FAO 3 Agro./Econ. National/Regional Quantitative 10yr 1yr? IMPACT / IFPRI 4 Climate/Agro./Econ. Regional/Global Quantitative 20yr. 6-24mo.? Forecast Accuracy World Modelers Comprehensive Subnational Quantitative 1-3yr 1mo. continuous evaluation Sources (click for web pages): 1: Famine Early Warning System, 2: Global Information and Early Warning System, 3: Organization for Economic Cooperation and Development/U.N. Food and Agriculture Organization, 4: International Food Policy Research Institute Approved for Public Release, Distribution A 3

4 Modeling at Sub-national Scales A simplified, generic model of some factors that affect food in. Edges represent affects/ affected-by and part-whole relationships. profits markets transport World Modelers will model at subnational scales, preserving the variability seen at smaller scales. This variability matters because food and other resources are distributed unequally, with consequences at subnational scales (districts or cities). Modeling variability is harder than modeling averages or trends. More factors must be considered. aid imports stock production shocks floods labor Will there be enough food to feed most of the world next year? Yes. Will Southern Sudan have enough food? This is a much harder question. El Nino/Southern Oscillation Approved for Public Release, Distribution A 4

5 How World Modelers Will Work Suppose the task is to forecast food in in districts of Southern Sudan two years out. Step 1: Generate or retrieve a generic qualitative, causal food model; Step 2: Modify the model for the specific analyses of Southern Sudan; Step 3: Build workflows of expert, quantitative models, where available; Step 4: Parameterize quantitative models and the qualitative, causal model; Step 5: Configure scenarios and run analyses, producing quantitative results for factors of interest (e.g., food, calorie intake); Step 6: Produce an uncertainty report that documents sources of uncertainty, run uncertainty-reduction procedures and sensitivity analyses; Step 7: Identify possible actions to affect factors of interest (e.g., peacekeepers at markets). Refine models based on Uncertainty Report Approved for Public Release, Distribution A 5

6 Step 1: Get a Generic, Causal, Qualitative Model By reading online material machines will build generic, qualitative models of food in. If such models exist (perhaps from previous analyses), machines will them. profits Sources and Technologies: markets transport Machine reading of online resources such as U.N. Food and Agriculture Organization Country Briefs and OECD long-term projections. aid imports stock production Machine grounding of entities in text into ontologies. Machine assembly of qualitative, causal models from fragments, based on Big Mechanism technology). shocks floods labor ENSO A generic model of food in * ** /agr_outlook-2016-en Approved for Public Release, Distribution A 6

7 Step 2: Tune Generic Model to Specific Analyses Decide which factors in the generic model are, or could be, relevant specifically in Southern Sudan, adding factors as necessary. Sources and Technologies: currency devaluation ethiopia kenya profits Machines can read recent reports about Southern Sudan to make good guesses about relevant factors. For example markets transport The spike in food at the end of 2015 coincided with the decision of the Central Bank to move from a fixed to a floating exchange rate regime that led to a devaluation of the local currency by about 84 percent. ** aid imports stock production Market activities have slightly improved in recent months in some -affected areas but food supplies remain well below the pre-crisis levels and food remain exceptionally high and volatile, largely influenced by the distribution of food aid. shocks floods labor Since the start of the in mid-december 2013, over 2.3 million people have fled their homes, including about 1.7 million ly displaced and about individuals currently hosted in neighbouring countries (Ethiopia, Uganda, the Sudan and Kenya) as s. Human analysts would edit the machine s choices based on expert or commonsense knowledge that machines won t have. ENSO nodes in the generic model considered relevant by the machine/analyst new nodes added to the generic model by reading about Southern Sudan ** Approved for Public Release, Distribution A 7

8 Step 3: Workflow Compiler Where quantitative models are available, link them to the qualitative causal model. This produces a computational workflow among models. Sources and Technologies: Build and maintain a corpus of expert, quantitative models in areas such as, weather, hydrology, forecasting, cropping and yield forecasting, market clearing and price setting, migration, etc. Build on technologies such as grid computing and model composition systems such as WINGS to suppport human-machine assembly of workflows of models. profits markets aid currency devaluation imports transport stock production FAO and OECD AgLink-Cosimo Model of Agriculture Markets ethiopia kenya Build on the natural language dialog capabilities of Communicating with Computers, which support human-machine construction of complicated biological models. shocks AgMIP labor DSSAT crop yield model floods ENSO NOAA Global Tropical SST Forecasts Blue boxes are extant software models, e.g.,, crop, and market models. Approved for Public Release, Distribution A 8

9 Step 4: Parameterize Models Find parameter values in online sources, set model parameters, and specify how outputs of models are combined at nodes in causal network. In effect, this transforms a qualitative causal model into a quantitative model. devaluation of local currency by 84% currency devaluation kenya profits transport markets Develop technology to help analysts find or write functions to combine numeric parameters when this isn t already done by numeric models. For example, while the FAO-OECD AgLink model combines production and import data to forecast, no numerical code exists to combine market, price and production data to estimate the degree of to food (in calories per person) in Southern Sudan. Such functions will often be described in literature (and could be found by machine reading) but might have to be implemented by hand. ethiopia = f(market,price,production) Sources and Technologies: Develop technology to semi-automatically online sources (e.g. maps), read data in various formats (e.g., Big Mechanism table reading of historic price data), and read parameter values in documents (e.g., 647,000 individuals ). FAO and OECD AgLink-Cosimo about Model of Agriculture Markets individuals aid imports stock estimated at shocks920,000 tons production floods labor DSSAT AgMIP crop yield model ENSO Approved for Public Release, Distribution A NOAA Global Tropical SST Forecasts Sources for numeric parameters (in green) include text, tables, and functions that combine outputs of model components 9

10 Step 5: Specify Scenarios and Run Analyses Set up scenarios. Scenarios are specified by input parameter values (blue arrows) that propagate through the model to yield results (red arrows). Run multiple scenarios. Parameter values may be sampled from distributions. This produces distributions over output values, confidence intervals and other representations of sensitivity or uncertainty. Sources and Technologies: The program will address some technical issues that continue to drive modeling research (e.g., running linked models that have very different time scales). Some performers may have capabilities to run very large simulations on supercomputers, clusters and grids. The program will develop technology to help analysts avoid semantic errors when they specify scenarios, not unlike some kinds of type checking or graph analysis in programming. For example, in the current model the analyst s distribution over might clash with NOAA forecasts. The analyst must be made aware of this and given options for, e.g., specifying that his distributions take precedence. aid profits markets currency devaluation imports ENSO transport JFMAMJJASOND JFMAMJJASOND shocks stock production NOAA Global Tropical SST Forecasts AgMIP labor DSSAT crop yield model FAO and OECD AgLink-Cosimo Model of Agriculture Markets ethiopia kenya JFMAMJJASOND JFMAMJJASOND Analysts can drive scenarios by having the machine sample parameter values from distributions, in blue (e.g. ) Approved for Public Release, Distribution A 10

11 Step 6: Analyze & Mitigate Sources of Uncertainty Generate an Uncertainty Report. Uncertainty accumulates along causal pathways, driven by input distributions, combining functions, model uncertainties, etc.... A single, scalar margin of error for analyses is not credible. Instead we want a detailed uncertainty report for each analysis. Currently is dominated currency by concerns. devaluation Future status is uncertain and effects on and markets profits are estimated to be large. Recommend analysis of multiple scenarios. markets transport ethiopia kenya Sources and Technologies: FAO and OECD AgLink-Cosimo Model of Agriculture Markets The program will develop technology to A) keep track of where uncertainty is introduced, B) trace its effects down causal pathways, C) apply uncertainty reduction methods at each point where uncertainty is introduced (e.g., ensemble models, biasing with or checking against historical data, asking an expert, optimum estimation methods, optimal sampling methods, etc.) D) run multiple scenarios to see the effects of uncertainty (sensitivity analysis). Uncertainty analysis is a human-machine task, but the machine can do much of the work of A,B,C and D under human guidance, and can do exhaustive diagnostic work (e.g., checking whether intermediate results are within expected ranges) that humans would find tiresome. In Southern Sudan, aid imports stock depend on. ENSO two years out are uncertain. shocks Recommend sensitivity analysis over ENSO. ENSO floods production NOAA Global Tropical SST Forecasts AgMIP labor DSSAT crop yield model Blue boxes represent illustrative contents of uncertainty reports. Approved for Public Release, Distribution A 11

12 Step 7: Identify Possible Actions Which actions are likely to have positive effects on factors? What can be done to reduce projected food in in Southern Sudan? Sources and Technologies: profits currency devaluation ethiopia kenya Adapt Big Mechanism algorithms that find drug combinations to combat cancer. These algorithms trace back from desired outcomes through causal models to find pressure points that probably have big effects. Because it is difficult for humans to find combinations of actions, we expect to adapt Big Mechanism algorithms like MUTEX to find cliques of pressure points where multiple actions can be effective. The program will build on human-machine, mixed-initiative planning algorithms that can take guidance from human users, as machines lack general world knowledge (e.g., reversing currency devaluation is harder than increasing food aid to camps) markets aid imports ENSO transport shocks floods stock Shortages are due primarily to, FAO which and OECD is due AgLink-Cosimo to and Model of at Agriculture markets. Markets We could increase at markets. We could also increase productionfood aid to s. NOAA Global Tropical SST Forecasts AgMIP labor DSSAT crop yield model Green nodes are influences or effects. The green box represents one of perhaps many plans to influence food shortages Approved for Public Release, Distribution A 12

13 Task Areas TA Description Supports Steps Builds On 1 Build Qualitative Models from Online Sources 1,2,4 Big Mechanism reading for causal fragments, grounding in ontologies, assembly algoithms 2 Workflow Compiler 3 Scientific workflow technology (e.g., WINGS), grid computing, distributed simulation, CwC collaborative dialog capability 3 Parameterize Models 4 Big Mechanism table reading, remote sensing feeds,crowdsourcing and polling 4 From Scenarios to Actions 5,7 Big Mechanism tech for finding pressure points in causal networks, AI planning tech for multi-step plans 5 Uncertainty Reports 6 CwC tech for human-machine examination of models will help. However, humanmachine uncertainty analysis for very complicated models requires new science Approved for Public Release, Distribution A 13

14 Test Problems and Phasing Mo.18 Mo.36 Mo.48 Develop Test Develop Test Develop Test Phase 1 Phase 2 Phase 3 In successive phases, the test problems become less like the development problems: Phase 1: The test problem is to analyze the same question as seen in the development period, but in a new country (e.g., food in has different causes in Venezuela and South Sudan so the models to be analyzed will be different). Phase 2: The test problem is to analyze a related question that uses some models and data sources from earlier problems (e.g., migration shares many factors with food, so a test might involve a migration problem when none had been seen during development); Phase 3: The test problem is to analyze a problem unlike any seen before, requiring the assembly of at least some data and models that haven t been seen in any previous development problems (e.g., via an MOU with OCIA (part of DHS) we might analyze the effects of continued on critical infrastructure in California). Approved for Public Release, Distribution A 14

15 Evaluation and Scoring Evaluation will test the central claims of World Modelers: It provides technology to enable analysts to rapidly build models to analyze questions relevant to national and global. Analyses will be comprehensive, causal, probabilistic, and timely enough to recommend specific actions that could avert crises. Evaluations will address specific questions about the value of the technology to analysts: Utility and Accuracy Can real analysts use the technology, and do they want to keep on using it? Are the models projections accurate enough to be useful, i.e., to support recommendations? Plausibility are analyses plausible, do they agree with concurrent, human-only, expert analyses (e.g., do we get roughly the same forecasts for Southern Sudan as experts at FAO, and if not, are they plausible enough to warrant analysts attention)? Diagnosticity and robustness do users understand the sources of uncertainty in their analyses, can they make analyses more robust with the help of the machine? Latency how long does it take to set up and run analyses? Approved for Public Release, Distribution A 15