Dimensions of Credibility in Models and Simulations Martin J. Steele, Ph.D. National Aeronautics and Space Administration (NASA) IT-C1 / Kennedy Space Center, FL 32899 Martin.J.Steele@nasa.gov June 2008
Overview Recent Historical Perspective The Idea of Credibility in M&S Dimensions of Credibility Concerns Findings Summary Final Thoughts 2
M&S Standard Development Mgt Decision Maker Interviews Pilot Studies CAIB Report Diaz Report NASA OCE Direction M&S Literature Interim Nov 06 Final Submitted Nov 07 2 Credibility Scales 1 New Credibility Scale External Efforts NASA-wide Formal Review 3
Recent Historical Perspective Outside NASA DoD M&S Validation (VPMM) M&S Quality DoE Predictive Capability Maturity Model (PCMM) ASME M&S V&V in Computational Solid Dynamics AIAA M&S V&V in Computational Fluid Dynamics Inside NASA The NASA Standards for Models & Simulations 2007 Summer Computer Simulation Conference Interim Version Development of NASA s Models and Simulations Standard 2008 Spring Simulation Interoperability Workshop Final Version 4
Goals for Excellence in M&S Goals of the M&S Standard Ensure that the credibility of M&S results is properly conveyed to those making critical decisions Assure that the credibility of M&S meets the project requirements Additionally, Form a strong foundation for disciplined development, validation and use of M&S Include a standard method to assess the credibility of the M&S presented to the decision maker 5
The Idea of Credibility in M&S
Development Foundation Accreditation Development Sargent, R. G. (c. 1980). Verification Validation 7
Operational Foundation Input Pedigree Results Operations Uncertainty Quant. Robustness Obtaining Credible Results Requires: Good Input Understanding of Uncertainty Sensitivity Analysis Accreditation Development Ver Val 8
Supporting Evidence Use History Supporting Evidence Model Mgt People Qual. Adding to Credibility: Past Use M&S Mgt Developer / User / Analyst Qualifications Accreditation Results Development Operations Ver Val Input Pedigree Uncertainty Quant. Robustness 9
Breadth to Consider Problem Statement System Understanding Modeling Choice Conceptual Model Validation Computational Model Verification System Analysis Validation Coding the Model Real World System System Understanding Detailed Understanding Validation Conceptual Model Verification Implementable Conceptual Model Accreditation Results Development Operations Supporting Evidence Ver Val Input Pedigree Uncertainty Quant. Robustness Use History Model Mgt People Qual. 10
Verification & Validation Verification Structure Flow Fidelity Validation: determining the degree to which a model or a simulation is an accurate representation of the real world How: Comparing to Conceptual Model Code Tracing Primitive Tests Min/Max Value Tests 12
Input: Source Notional Subject Matter Expert Applicability to current problem Referent Quality relative to current problem Referent System Referent Environment Input Pedigree Input Form: What s the character of your analysis? Average Uniform Triangular Quantity Curvilinear Authoritative Data 13
Accuracy & Uncertainty Accuracy: True Value Modeled Value Uncertainty in Modeled Value Uncertainty in True Value Uncertainty: Sources Size How Confident Epistemic Aleatory Reducible Irreducible Subjective Variability Model Form Inherent Incomplete Information Stochastic 14
Robustness Robustness of Results, i.e., Sensitivity of: The Real World System (RWS) The M&S Insensitive To Changes M&S Worst Situation M&S shows a Robustness not present in the RWS - Validation Issue - M&S not so useful Best Situation RWS is robust (Insensitive to Changes) & the M&S matches the RWS Better Sensitive To Changes OK Situation RWS is sensitive to change & the M&S matches the RWS Not a Good Situation M&S is not robust, but RWS is - Validation Issue - Results will be overly conservative Sensitive To Changes RWS Insensitive To Changes 15
Use History & Management Use History: Similarity of Uses Analogous Systems Exact Systems Length of Time in Use Just Developed Just Updated Long-Term Successful Use M&S Management: Models & Data under Configuration Control Models are Maintained Sustained 16
People Qualifications & Tech Review People Qualifications: Education Training Experience In M&S With the Modeled (Real World) System Technical Review: When accomplished During M&S Development During M&S Operations Ver Development Val Input Pedigree Operations Uncertainty Quant. Qualifications & Independence of the Peer Review Group: Self Internal Organization External Non-Expert to Expert Robustness 17 Level of Formalism Planning Documentation
When to Apply the Standard?
Scope of the M&S Standard Standard covers the use of M&S affecting: Critical Decisions Human Safety Mission Success As defined by each Program 19 The focus is on M&S for flight and ground support projects Operations Test & Evaluation Manufacturing and Assembly Design and Analysis M&S Results Influence 4: Controlling 3: Significant 2: Moderate 1: Minor 0: None IV: Negligible III: Marginal II: Critical I: Catastrophic Project Consequence Sample Risk Matrix
Basic Ideas Documentation of M&S Activities (Sections 4.1 4.6) Credibility Assessment (Section 4.7 & Appendix A) Reporting to Decision Makers (Section 4.8) M&S Analysis Results A statement on the uncertainty in the results Credibility of M&S Results Identify Unfavorable outcomes Violation of assumptions Unfavorable Use Assessment Difference Between V&V & Use Assessment 20
Use Assessment Expected Output Range Validated Output Range Validated Input Domain Intended Input Domain 21 Note this is a 2-dimensional example of a potentially multidimensional input domain & multi-dimensional output range
Sommaire & Future Directions
Future Directions Track 1 Internal Deploy Use of Standard / Collection of Data on Use Assess Revise Track 2 M&S Discipline Specific Develop Guides (RPGs, Handbooks) For each M&S type in relation to the NASA M&S Standard Track 3 External Collaboration Other M&S Standards/Guides Professional/Academic Organizations 23
Broadening the Perspective of M&S V&V and Simplifying It!
Network Layered Protocol Approach Like the Layered Network Protocol Model 25
Layered M&S View (Influences in M&S Results) M&S V&V and Credibility Assessment User Input including Run Setup M/S Input M/S Output Model / Simulation Application Software Analyzing Output including Post-Processing of Output Data Industry Standards and Broad Use Operating System Software Computer Hardware Need for a Clearinghouse for Commercial & Open Source M&S Languages & Application Software 26
Final Thoughts
Concerns & Findings General Concerns Broad Applicability Subjectivity Software Similarity The Analysis vs. The People If we don t have this Standard Many varieties of Credibility Assessment will be are being presented Findings Yet Another Standard Terminology (Denotations & Connotations Types Purposes Education, Experience, & Training 28
Why doesn t/can t the Software Engineering Req ts do the job? Models and simulations may be implemented in software, but they encompass much more than software engineering The SW Engineering NPR (7150.2) does not address many issues critical for M&S, e.g., development of models validation against experimental or flight data uncertainty quantification operations and maintenance of M&S Models are sometimes developed in hardware Physical Models (Scaled) Analog Models Software Engineering Models & Simulations 29
Comparison Software Performs a task within a system Purpose performance of tasks/functions for a system Requirements are functions the s/w shall perform More discrete functions in the code (function-centric) Software shall do this S/W shall do that Etc. Traditional s/w functions either work or don t work Typically, there is no uncertainty M&S Provides a representation of a system Purpose Analysis or Representation of a system or of system functions for insight Requirements are behaviors the simulation model shall exhibit System behavior-centric Includes more assumptions how different parts of the simulation model work Typically includes uncertainty in the behavior of the simulation Totally focused on Software Development Must include requirements (& reporting) on M&S Use 30
Software & M&S You can follow software standards correctly in model development BUT STILL produce inaccurate or missleading results Software Standards do not deal with Software (or Model) use! 31
Final Statements Communicating Results Framework for Discussion of Results Credibility Credibility Factors Beyond VV&A Accreditation Results Development Operations Supporting Evidence Ver Val Input Pedigree Uncertainty Quant. Robustness Use History Model Mgt People Qual. Technical Review With dimensions composing each factor of M&S Credibility 32 It is: High Level Broadly Applicable (To All M&S Types!)
Thank You