HR001117S0017 World Modelers Frequently Asked Questions As of April 21, 2017 Q48. We were wondering if we can focus on a few specific Technical Areas mentioned in the BAA. In our case, if we primarily concentrate on two TAs, can we expect to receive developed models from another funded performer working on one of the other TAs or from DARPA? If not, we shall enhance our team accordingly? A48. Proposers do not have to necessarily propose against multiple TAs or particular combinations of TAs. Again, proposers should strive to propose the most highly capable team necessary to develop an approach to at least some, but not necessarily all, the technical challenges of the program. Proposers may provide proposals addressing multiple or a single task area. However, proposers should not overreach and propose to a greater number of task areas than their approach and expertise level of their teams can bear. Much like Big Mechanism and Communicating with Computers, this program will involve collaborative technology development among performers. Teams may be proposed, or alternatively, performers funded to work on specific Task Areas may be assembled into teams after award at kickoff. Proposers may identify quantitative models that they work with (e.g., a proposer who models groundwater might bring groundwater models to the program) but no one is required to supply quantitative models. DARPA will collect and provide quantitative models to funded performers. Q47. During proposer s day and in the BAA there is reference to leveraging capability from Big Mechanism. Big Mechanism is not yet available on the DARPA Open catalog. Can you provide information (to include documentation, code, etc.) on capability we should leverage from Big Mechanism? A47. Access to capabilities from Big Mechanism is not a necessary condition for developing a responsive proposal to the World Modelers BAA. Big Mechanism is still an active program with development efforts still underway. Finalized models, deliverables, and/or publications will be made available via the DARPA Open Catalog https://opencatalog.darpa.mil/ World Modelers will kickoff as its own freestanding program. When Big Mechanism ends, if there are technologies or capabilities that can be leveraged, then performers may do so.
Q46. As to temporal scale, World Modelers is committed to evaluating the accuracy of its forecast, so use cases that forecast one to five years into the future are preferred. [emphasis added] Can you say more about the evaluation? For example: Should we also expect sub annual forecasts that are used in annual program / performer evaluations? Will the program adopt uniform metrics (e.g. Brier score, log loss, hit rate, etc.) or is this up to the performer? Will any forecasts have to be delivered regularly to the evaluator? Is there a provision for evaluating long term forecasts? A46. Ultimately, World Modelers does not insist that all forecasts be accurate or that all forecasts be quantitative, but that analysts should know when they can trust quantitative forecasts. However, the models forecasts should be accurate enough to be useful, i.e. to support recommendations. Directly from the BAA, note that the accuracy of forecasts is grouped with utility, because models can be useful even when they are not very accurate. Nevertheless, the World Modelers program will work in domains that afford opportunities to continuously track the accuracy of forecasts. For example, migration is an ongoing process, so at any moment one can ask for a forecast of the number of people who move from one location to another during an interval of time, and at the end of the interval one can test the accuracy of the forecast. Migration is a good domain because it affords thousands of such tests, in contrast with analysis questions about one off events (e.g., tsunami). The methodology for evaluation will be very clear and shared among all funded performers. The specific details or approach to evaluation will be driven to some degree by the specifics of the use cases and stakeholders interested in the program. A government evaluation team will develop World Modelers use cases for development and testing in each phase. The development plan for the World Modelers program is to build and refine complete World Modelers systems continuously, as opposed to waiting to integrate component technologies until they mature. Proposers are welcome to suggest use cases. However, to help ensure that World Modelers technology is general, these problems must be made available to all participants and all participants must solve the test problems set by the evaluation team. These requirements will help to avoid situations in which participants work in isolation and develop problem specific technologies. Q45. I am interested in talking with other potential proposers to explore teaming opportunities. Where can I find the list of attendees at the Proposers Day? A45. The link to the World Modelers attendee list, which includes those who agreed to share their contact information, is publicly available here: https://www.schafertmd.com/darpa/i2o/worldmodelers/pd/content/downloads/worldmodelers _roster.pdf Q44. The government entities clause in the CFP indicates that for the government to compete, it must provide written justification to demonstrate that the work is not otherwise available from the private sector. However, if another company/organization is the prime, and they establish an agreement with the government to be a subcontractor part of their team, is the 'demonstration that the work is not available' still applicable?
A44. The rules for Government entities apply both if they act as a prime or subcontractor. The BAA identifies both legal citations as potential justification starting points or citations that are not acceptable. The proposer should discuss the issue with their GC to determine if they have any legal exceptions that would allow them to receive funding from another Government agency. Q43. I would like to provide a briefing or white paper and meet with the PM to discuss my approach. How can I make these arrangements? A43. During the BAA proposal timeframe, DARPA cannot legally comment on aspects of World Modelers outside of what is within the BAA and in the open Q&A document available to all proposers. Q42. I read through the BAA and plan to organize a team to submit a proposal. Due to the nature of focusing multiple domains, this effort could be relatively large. May I ask if the program has some expected level of funding for each award? Of course, I understand that budget realism is a must, but would like to have an idea on how to shape the team. A42. DARPA cannot provide budgeting guidance or comment on specific budgetary questions from potential performers. Since each proposal has its own unique statement of work and may or may not fully propose tasks from a TA, DARPA has no up front method to uniformly determine what is the appropriate level of funding. Thank you for your understanding. Q41. Unfortunately, I missed the World Modelers Proposers Day that occurred on February 3. Were there minutes recorded from that event? If so and if appropriate, can you guide me to WM information that has emerged from that day or since? A41. The Question and Answer portion of the Proposers Day was captured and has been made available as a document to all potential proposers here: http://www.darpa.mil/work with us/opportunities Q40. I have a question about the interface between the Technical Areas of World Modelers. How can proposers for individual TAs go about assuming the type of model (in terms of representation) that will be made available to them, or that they will need to make available? A40. Proposed approaches do not necessarily need to focus on the exact format or content of the outputs of the models. As referenced in the BAA, domain experts and computer scientists have already integrated nine quantitative models to evaluate 20 qualitative scenarios. It is up to the proposers to address how to do this type of analysis in a less time consuming manner. Integrating and parameterizing these and other models potentially developed under World Modelers in the future should be done in a single computational framework. The aim of World Modelers is to automate much of the work done by humans so as to conduct such analyses in roughly one month, not necessarily constrained by the types of quantitative and qualitative models being integrated Selected performers will be required to collaborate with each other after award and kickoff. More specifically, performers working among different TAs will ultimately develop products that will be integrated in a single computational framework, though not necessarily immediately upon kickoff.
Proposers should carefully scope what they can contribute to the World Modelers program and should not feel obligated to propose complete World Moldelers systems. Proposers should be clear about potential or anticipated dependencies and synergies between what they propose and other technologies or TA approaches. As of April 11, 2017 Q39. Will political considerations (e.g. elections, social uprisings, terrorist acts, etc.) be of interest to the program? A39. Yes. However, DARPA is aware that socio political models are less mature and accurate than, say, geophysical or biophysical models. They tend to be qualitative, and even the quantitative ones tend to be retrospective rather than prospective. Where social phenomena are modeled, the models should stay close to the measurable causal antecedents of social phenomena (e.g. economic and physical security as antecedents of decision to migrate), so we can back off to forecasting antecedents if the social models are inadequate. Mechanistic, ideally quantitative, models are the goal, and if social phenomena cannot be modeled mechanistically and quantitatively, then the program will make do with models of causal antecedents of these phenomena. Q38. How big should my team be? How big can it be? A38. Proposers should strive to propose the most highly capable team necessary to develop an approach to at least some, but not necessarily all, the technical challenges of the program. Q37. What is the program budget or anticipated award? A37. DARPA is not sharing this program s budget information with proposers. The focus of the proposer should be on addressing the World Modelers Task Areas and objectives. Concerning awards, multiple awards are anticipated. The level of funding for individual awards made under this solicitation has not been predetermined and will depend on the quality of the proposals received and the availability of funds. Q36. What should be complete by the end of year one? A36. This program seeks to have a fast start in that a usable model (not necessarily perfected) should be developed to address the food insecurity problem. Expertise will not be explicitly excluded, but expertise is not the only way to get model development going quickly. Q35. What will be the user role in running the model? Will users be in the loop or out of the loop? A35. This is up to the proposers. However, machines will not solve modeling problems autonomously.
Q34. Will one of the deliverables be code? A34. Yes, DARPA anticipates deliverables will include code. The deliverables need to run on authentic analytical problems. Q33. Can the causal model be built from information or data that is mined? A33. That is an approach, but certainly not the only approach. Q32. What are the best models or databases available? A32. DARPA is not aware of the best and this may be dependent upon approach and model design. DARPA is open to proposer ideas. Q31. Can we talk to the agencies that are interested in World Modelers? A31. DARPA does not wish to impede communications that might help the World Modelers program, but no such interactions are necessary for proposers. Q30. To get qualitative and quantitative models in sync seems to be a big technology problem. How will World Modelers address this issue? A30. Correct. Getting multiple models in synch is an open research problem. It isn't just qualitative and quantitative models that need to be synched, but also other varieties of models, such as regression models and ODE models and systems dynamics models. The general problem is one of maintaining multiple representations of roughly "the same thing," representations that might make slightly different modeling assumptions, or work at slightly different temporal or spatial scales. Humans manage to translate the outputs of one model into inputs of another, or reconcile outputs of ensembles, but these are often semantic judgments. It remains to be seen whether machines can have sufficiently rich semantics of models to perform these sorts of judgments. Q29. How will World Modelers use sensitivity analysis approaches? A29. Machine assisted refinement of models as a result of sensitivity analyses would be of interest to the World Modelers program. Identification of good proxies for parameters would similarly be of interest. Q28. Will abstracts be requested before proposals? A28. No, there will be no request for abstracts.
Q27. When one thinks of a model of the world, there may be lots of variables and broad strokes. How can these all be captured or characterized? A27. All modeling involves tradeoffs. One model might be more realistic than another in the sense of representing more true causal factors, but it might be only slightly more accurate in its predictions and considerably harder to explain. Additionally, the more realistic model might have less data available to estimate its parameters, or it might over fit its data. Yes, World Modelers is trying to model complicated systems, but the modeling challenge for World Modelers technology is utterly familiar: How to build useful models that, for various reasons, don't represent all causal factors. Q26. How important is the run time of a World Modelers model? A26. The run time is less important than the usefulness of analysis graphs, demonstration of causal relationships, and comprehensive understanding of factors related to uncertainty. All things being equal, run times of weeks or months would be a concern. World Modelers seeks to facilitate analyses that currently take humans weeks or months. Run time in and of itself is not a strong criterion for evaluating World Modelers technology. Q25. One can imagine a large library of models will be needed to run a successful World Modelers program, how will these be assembled? A25. DARPA will assemble a library of models, at least for the first year's use case. Another interpretation of this question focuses on the verb "assemble." Assembling models will require that they "talk" to each other, which might involve, among other things, ontologies to map outputs to inputs and so on. Assembling quantitative models in support of analyses is one of the central problems in the program. Q24. Will proposals that include Subject Matter Experts in areas such as agriculture, social science, or finance be viewed favorably? A24. This is principally a technical program to facilitate different types of analysis using causal models. However, Subject Matter Experts (SMEs) might guide the development of analyses of national and global security questions, and, of course, they will have domain knowledge that machines might lack. That said, if the sole purpose of SMEs is to ensure that analyses are credible, then their value to the program is somewhat limited because the program is intended to develop modeling technology, not necessarily high quality analyses. One can imagine SMEs serving as stand ins for users of the technology, or as authentic users who intend to bring the technology into their own workplaces. But it is also conceivable that World Modelers technology can be built without the assistance of SMEs. Q23. What is the role of machine learning in World Modelers? A23. Machine learning can have many roles in the program, from learning proxies for quantitative model parameters to learning lightweight approximations to expensive quantitative models. Solutions might involve machine learning concepts such as ensemble modeling and feature
selection. It's really for proposers to say how machine learning will figure in their technical proposals. However, World Modelers is not simply an exercise in data mining. Its goal is humanmachine analysis based on causal models. Q22. How separate or separable are the Task Areas? A22. Proposers may provide proposals addressing multiple or a single task area. However, proposers should not overreach and propose to a greater number of task areas than their approach and expertise level of their teams can bear. Much like Big Mechanism and Communicating with Computers, this program will involve collaborative technology development among performers. Teams may be proposed, or alternatively, performers funded to work on specific Task Areas may be assembled into teams after award at kickoff. Q21. Is the intent to build generic models for any domain? A21. Yes and no. Yes, we are interested in modeling technologies for many domains or use cases. But we are interested in analyzing specific questions (e.g., about food security in South Sudan) so generic models are useful only to get the specific analysis started. Q20. What will be the use cases? A20. There will be common program use cases, as well as potentially performer suggested use cases. However, all performer suggested use cases must be made available to all funded World Modeler performers, and all performers will also have to deliver a model and output for a common use case provided by DARPA and the government evaluation team. We want to avoid a situation where a particular performer works in isolation on a private use case. More importantly, we want to avoid a situation where modeling technology is useful only for a private use case. Q19. There are powerful models available, but they are proprietary. How can the program overcome potential limitations? A19. Since this is a fundamental research program, it is preferable for the models to build upon open source models available to the public. In Phase I, the emphasis will be on humans and machines working together to assemble qualitative and quantitative analyses; the quality of the quantitative models will be secondary. If proprietary models are part of a solution, the proposal should address how to accomplish these goals and overcome any licensing restrictions. Q18: Is there a specific Task Area for the user interface component? A18. Task Areas four and five encompass work (configuring the model to run and uncertainty analysis) that will involve the interface. There is no one Task Area that just addresses the user interface on its own.
Q17. Where does interface design and building fit in this program? A17. This is an important part of World Modelers, much like the DARPA Communicating with Computers program. Humans and machines will communicate ultimately to run the models. The interface will be an important consideration for the uncertainty reports under Task Area 5. Proposers may employ an existing interface technology or build their own. However, some aspects of World Modelers can be addressed without worrying about interfaces with humans, so proposers should not feel obliged to explain how machines and humans will interact unless this is genuinely a necessary part of the work they propose. Q16: What is the expectation for the quantitative models to change as conditions related to national and global security change? A16. It is likely that structural changes to quantitative models developed over years or decades will be rare, as many of these models are tested and trusted. However, the parameters impacting the model inputs are likely to change (e.g. soil erosion, the anticipated impact of El Niño in a given year, or disruptive change like a nation state dividing). Q15. How will component models, such as crop models, be run when there is little soil data available for a region of interest? A15. The expectation is that proposers will address data needs or develop approaches that accommodate lack of complete data. Q14. What role will crowd sourcing play in terms of model parameterization? A14. Crowd sourcing may be considered as one of many potential sources of information or as proxy for data needs of quantitative models. Q13. Will DARPA Big Mechanism or related program technologies, models, or results be provided? A13. Yes, to the extent possible, as Big Mechanism is still an active program with development efforts still underway. Finalized models, deliverables, and/or publications will be made available via the DARPA Open Catalog https://opencatalog.darpa.mil/ Q12. What are the transition goals for World Modelers? A12. This is considered a fundamental research or basic research program, but the goal remains delivery of useful models and modeling technologies for interested stakeholders.
Q11. Will the analyses be expected to consider global warming effects? A11. That is a possible part of the solution, but this does not mean that World Modelers will model the climate. Rather, it is envisioned the World Modelers program will treat climate effects as drivers of downstream processes; it will focus on how climate change may drive other processes, and it will model climate in terms of driver variables without addressing climate change in and of itself. Q10. What are the types of questions that are expected to be answered by the models developed in this program? A10. The focus will be on questions related to national and global security (e.g. migration patterns, predictions of food riots, etc.). Food security will be the first focus, since models addressing this challenge are fairly well characterized. At the end of the final Phase III of the program, the objective is for a user to ask any type of question related to national and global security in any particular region of the world. However, for example, the model will not be asked questions that have no obvious security implications, such as who will win the Grammys?. Q9. Will the models be expected to filter fake news? A9. This is not a program about fake news. That said, the expectation is that a thorough analysis would have a strong enough internal logic and gather enough data to identify anomalies. Q8. Will quantitative or qualitative models be provided? A8. Qualitative models should be developed by a machine. An educated reader of news or expert analyses could generate a qualitative model, but this can and should be done by machines, to advance the state of the art. Proposers may identify quantitative models that they work with (e.g., a proposer who models groundwater might bring groundwater models to the program) but no one is required to supply quantitative models. DARPA will collect and provide quantitative models to funded performers. Q7. Will data, such as United States Agency for International Development pricing data, be provided for the models? A7. It is anticipated that data will be provided for the first year of Phase I in order to catalyze model development. Q6. What is the anticipated role of social media in this program? A6. Parameterization of qualitative models involving human behavior is an anticipated challenge. Proxies may involve social media, satellite imagery, or other approaches that may be detailed in proposals. DARPA is open to imaginative but feasible approaches.
Q5. What is the latency target for the program? A5. Complicated analyses currently take six months to two years, and the output target for the program is within the range of weeks to one month with the analyses being at least comparable to, if not more useful than, those currently generated entirely by humans. Q4. How much should computers engage with humans in terms of data curation? A4. The expectation is that World Modelers will involve human and computer collaboration, given the differing strengths of each. For example, development of qualitative models has generally been done by humans, but setting the parameters of quantitative models could likely be done through machine reading. Q3. What will be the anticipated output of World Modelers to a specific question? A3. It will not just be a number or a statement. Each qualitative and quantitative model component will conduct its own analysis. Via the uncertainty report, there will be an indication of what is unknown and uncertain, and, ideally, why. This is a new area of research, and it is anticipated that the outputs will be similarly nuanced. In order for the output to be useful, its limitations should be called out. Q2. What is an uncertainty report? A2. This is new territory. Iterative model development and incorporation of numerous quantitative models may introduce and propagate uncertainty. It is important for the output to be useful that this uncertainty be characterized. This is the goal of Task Area 5. Sources of uncertainty should be identified. Ultimately, World Modelers does not insist that all forecasts be accurate or that all forecasts be quantitative, but that analysts should know when they can trust quantitative forecasts. However, he models forecasts should be accurate enough to be useful, i.e. to support recommendations. Q1. How is World Modelers different from other DARPA programs, such as Explainable Artificial Intelligence (XAI), Data Driven Discovery of Models (D3M), and Probabilistic Programming for Advancing Machine Learning (PPAML)? A1. There is certainly a common theme in all these programs. They all emphasize causal models sufficient for explanation. Prediction is good, but causal explanation is better. The interest of World Modelers is in solving problems, such as the food insecurity example, which involve heterogeneous, interacting systems on variable spatial and temporal scales. In the case of World Modelers, the causal processes are slow moving (in terms of months to years in the cases of weather patterns, migration, human behavior, etc.). Additionally, the focus of World Modelers is upon causality and mechanisms, not big data in and of itself. World Modelers will probably rely on some statistical models, occasionally, but, ideally, these will have causal interpretations. The output should incorporate both quantitative and qualitative analyses.