Generic Methodology for Verification and Validation Overview, Applications and Lessons-Learned

Transcription

1 ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th 2013 Generic Methodology for Verification and Validation Overview, Applications and Lessons-Learned

2 Background Global Military Trend Simulation Increased Joint Operation More Interoperability & Reuse ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

3 Background Military V&V Issues Different Processes & Products Different Domains & Applications Different Interpretations & Concepts Different Maturity Levels Different Techniques & Tools ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

4 GM-VV Generic Methodology for Verification and Validation Initiated and Support by WEAG REVVA projects as the basis; evolved into NATO-STO MSG-073 Key Objectives Methodology for V&V that embraces a wide variety of M&S technology and uses Common basis for V&V cooperation, information sharing and tools Standardization ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th 2013

5 GM-VV Generic Methodology for Verification and Validation Provides Framework of Generic V&V products, processes and organizational components Application guidance by example, tools and techniques catalog Tailorable Structured Risk-driven Argumentation Evidence-based ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th 2013

6 GM-VV Technical Framework: Focus on M&S Quality A quality system is fit for the users purposes, provides the needed features, and contains few, if any, important deficiencies - J.M Juran, Quality is Free, 1988 ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

7 GM-VV Technical Framework: Quality Properties M&S hardware and software/code Correctness Utility AND Fidelity M&S Abstracts and Approximates Reality ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

8 GM-VV Technical Framework: Risk-Based Decision-Making Good and Defensible M&S Decisions Understanding the limitations in modeling and simulation quality, and the implications (cost, safety, time, effectiveness) for its users ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th 2013

9 GM-VV Technical Framework: Uncertainty and Evidence Objective Demonstrate with sufficient confidence that model or simulation fits the intended use and doesn t pose unacceptable risks; to support well-informed decision making Requires Right Evidence Credible Useful Accurate Auditable Traceable Initial Uncertainty? V&V Evidence Residual Uncertainty? ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

10 GM-VV Technical Framework: Business-Driven V&V Approach Seeking a balance between Risk level and tolerances AND Costs and available resources V&V Rigor (Investment) low high Risk Level ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th 2013

11 GM-VV Technical Framework: Risk-Based V&V Strategy Component and Integration V&V System V&V and Acceptance Likelihood of M&S quality defects (Technical Risks) high low medium Level 3 Level 2 M&S SoI Quality Property (i.e. validation metrics) Level 1 Level 2 Level 4 Level 5 Level 3 Level 4 Level 3 low medium Impact of M&S quality defects (Business Risk) high ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

12 GM-VV Technical Framework: Argumentation Approach Acceptance Goal Argument Acceptability Criteria Context Information Context information M&S intended use (or domain of use) M&S use risks (Impact!) Other relevant M&S and V&V context information Acceptability Criteria M&S SoI Quality Properties (validation metrics) Conformance Tolerance Levels (if possible) Uncertainty (initial) and risk priority ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

13 GM-VV Technical Framework: Argumentation Approach Acceptance Goal Argument Acceptability Criteria Context Information Context information V&V metrics and techniques knowledge (e.g., rigor, cost) Reference (experimental/sme) data availability Other relevant V&V and M&S context information Evidence Solutions Evidence solutions Specific V&V technique Design of experiments Test-case specification Reference data ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

14 GM-VV Technical Framework: Argumentation Approach Acceptance Goal Argument Implement and execute BUT... it never works out as planned or expected! Acceptability Criteria Evidence Solutions V&V Results Outcomes itself Log execution and variances Express uncertainties and risks V&V Experimental Frame V&V Execution V&V Results ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

15 GM-VV Technical Framework: Argumentation Approach Acceptance Goal Argument Context Information Acceptance Claim Argument Acceptability Criteria Acceptability Claims Evidence Solutions Items of Evidence V&V Experimental Frame V&V Execution V&V Results ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

16 GM-VV Technical Framework: Argumentation Approach Acceptance Goal Context Information Acceptance Claim Context Information Argument Argument Acceptability Criteria Acceptability Claims Evidence Solutions Items of Evidence V&V Experimental Frame V&V Execution V&V Results ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

17 GM-VV Technical Framework: Organizational and Managerial Levels Effective V&V requires Proper project organisation Proper project management Internal quality control Efficient V&V projects require Enterprise support Enterprise Level Project Level Technical Level Knowledge Repository Maturity, knowledge and tools ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

18 GM-VV Implementation Framework Overview Generic Components Phase 1 Instantiation Phase 2 Risk-Based Optimization Enterprise Level P O P Tailor Actions Permanent V&V Organization Tailor Actions Project Level P O P Tailor Actions V&V Project V&V Project V&V Project V&V Project Tailor Actions Technical Level P O P Tailor Actions Technical Technical V&V Technical V&V Technical Approach V&V Approach V&V Approach Approach Tailor Actions ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

19 V&V Requirements Definition V&V requirements V&V context Information V&V user/sponsor needs V&V Agreement Acceptance Planning V&V argumentation structure V&V context information Project Memory V&V requirements V&V context information V&V Planning V&V plan V&V context information V&V argumentation structure V&V requirements V&V context information V&V argumentation structure V&V Execution V&V experimental frame V&V results V&V context Information V&V argumentation structure V&V requirements V&V context information V&V plan V&V argumentation structure V&V Assessment and Integration V&V argumentation structure V&V user/sponsor needs V&V requirements V&V context information V&V plan V&V experimental frame V&V results V&V argumentation structure Acceptance recommendation V&V requirements V&V context information V&V plan V&V results V&V argumentation structure V&V requirements V&V context information V&V argumentation structure Acceptance Assessment and Integration V&V Product Delivery Acceptance recommendation V&V argumentation structure V&V report ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

20 V&V agreement V&V project status report V&V enterprise memory Project Assessment and Control V&V project status report V&V project plan Decision Management V&V agreement V&V project status report V&V enterprise memory Risk Management Project Planning V&V agreement V&V project plan V&V project status report V&V enterprise memory Configuration Management Technical Processes Information Management V&V project plan V&V project status report Measurement V&V project status report V&V project plan Project Memory ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

21 V&V agreement V&V project memory Project Portfolio Management V&V user/sponsor needs V&V agreement V&V project memory Life-Cycle Model Management V&V Agreement Management V&V agreement V&V project memory Resource Management Project Planning V&V project memory V&V products Quality Management V&V agreement V&V products Enterprise Memory ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

22 GM-VV Documentation and Status Official SISO Material Useful Reference Roza, M. et. al. The Generic Methodology for Verification and Validation to support acceptance of models, simulations and data. The SCS Journal of Defense Modeling and Simulation: Applications, Methodology, Technology, October 25, 2012 ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

23 GM-VV Application V&V Studies for the Dutch Ministry of Defense A Brief Overview, for more technical details Roza M. et.al. GM-VV illustrated: an educational example from the human driving behavior research domain. SISO Fall-SIW, Orlando, 2010 Voogd J. et.al. The GM-VV tailored for a naval ship-handling training simulation. SISO Euro-SIW, The Hague, 2011 Roza M. et. al. The development, verification and validation of a distributed air operations mission training environment. AIAA MST Conference, Boston, 2013 Voogd J. et. al. Implementing the GM-VV enterprise, project and technical levels. SISO Fall-SIW, Orlando, 2013 ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

24 V&V Study with GM-VV Heavy Weather Ship Handling Simulator Is simulation valid for training and doctrine? Key V&V benefit Defensible ship dynamic model and motion system acquisition decision ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

25 V&V Study with GM-VV Distributed F16 Operation Familiarization Simulation Is a valid alternative? Key V&V benefit Better model, simulator and training requirements ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

26 V&V-Study with GM-VV Public Order Management Serious-Game Key V&V benefit Maximized training benefits How to improve fidelity and training? ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

27 V&V-Study with GM-VV Fighter Aircraft Robust Power Management Key V&V benefit Currently in progress Model replicative and predictive fidelity? ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

28 GM-VV Application Our lessons-learned Pro s Flexible, versatile and widely applicable Scalable and tailorable Good structure and guidance Facilitates high quality processes and results BUT... Requires initial work before use (instantiation!) Specific application/domain knowledge, methods and techniques (SME!) Abstractness and steep learning curve ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

29 GM-VV Application Closing the gap through the enterprise level Permanent V&V organization Ready to use method instance (V&V life-cycle models) Catalog with application and domain specific V&V, risk and uncertainty analysis techniques, Acceptance criteria and validation metrics, Argumentation language and patterns Consolidation and leverage of lessons-learned Trained V&V professionals and tool-suite ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

30 Questions Generic Methodology for Verification and Validation ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

31 More Questions information Generic Methodology for Verification and Validation Generic Methodology for Verification and Validation Dr. Manfred Roza, M.Sc. GM-VV Drafting Group Editor for Vol. 1 and Vol. 2 REVVA2 and NMSG073 Technical Lead Dr. Jeroen Voogd jeroen.voogd@q-tility.nl GM-VV Drafting Group Editor for Vol. 3 REVVA1, REVVA2 and NMSG073 Member ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th

32 Acknowledgment Q-tility background, sponsoring and development Initiated by Dutch MoD Consolidation and exploitation of the developed V&V knowledge Need for permanent, independent and efficient V&V services Sponsored by the Dutch MoE&I Supported by the NATO-STO M&S Group ASME 2013 V&V Symposium, Las Vegas, May 22 nd -24 th