Practical and Technical Challenges in Verification and Validation

Transcription

1 Practical and Technical Challenges in Verification and Validation William L. Oberkampf, PhD Sandia National Laboratories (retired) Consulting Engineer Austin, Texas American Society of Mechanical Engineers Verification and Validation Symposium Las Vegas, Nevada May 2 4,

2 Outline History of the development of V&V standards Ideas for resolution of conflicting concepts Technical challenges in modeling and simulation V&V Conclusions 2

3 Goals of Verification and Validation Assessment and improvement of the credibility, accuracy, and trustworthiness of products or services Assessment procedures: Principles and procedures must be applicable to broad classes of products or services Must be able to make objective measurements of the trustworthiness of contributing elements Improvement procedures: Procedures must be applicable to contributing elements, subelements, sub-sub-elements, etc. Procedures must be relevant to the trustworthiness of the product or service 3

4 Institute of Electrical and Electronics Engineers IEEE: Standard for Software Verification and Validation Plans Provided minimum requirements for software V&V plans Definition-Verification: The process of evaluating the products of a software development phase to provide assurance that they meet requirements defined for them by the previous phase Definition-Validation: The process of testing a computer program and evaluating the results to ensure compliance with specific requirements IEEE: Guide for Software Verification and Validation Plans Recommended approaches for improved software V&V planning 4

5 American Nuclear Society ANS (Renewed 1998): Guidelines for the Verification and Validation of Scientific and Engineering Computer Programs for the Nuclear Industry Recommended guidelines for V&V for scientific and engineering computer programs Definition-Verification: The process of evaluating the products of a software development phase to provide assurance that they meet the requirements defined for them by the previous phase Definition-Validation: The process of testing a computer program and evaluating the results to ensure compliance with specified requirements Guidelines based on a software V&V perspective 5

6 U. S. Department of Defense In the early 1990s the Defense Modeling and Simulation Office (DMSO) was tasked to study V&V concepts Are the concepts of V&V established by IEEE appropriate for DoD needs? In 1994, fundamentally different concepts were codified: Definition-Verification: The process of determining that a model implementation accurate represents the developer s conceptual description of the model Definition-Validation: The process of determining the degree to which a model is an accurate representation of the real world form the perspective of the intended uses of the model The emphasis shifted away from software reliability, to modeling and simulation credibility. 6

7 American Institute of Aeronautics and Astronautics In 1992 the AIAA Committee on Standards for Computational Fluid Dynamics began studying varying terminology and concepts of V&V In 1998 the Committee produced the first engineering standard for V&V based on M&S concepts: (AIAA-G , renewed 2009) Guide for the V&V of Computational Fluid Dynamics Simulations Definition-Verification: The process of determining that a model implementation accurately represents the developer s conceptual description of the model and the solution to the model Definition-Validation: same as DoD definition Codified the concepts of: Importance of solution verification Validation hierarchy Prediction distinguished from validation 7

8 American Society of Mechanical Engineers In 2001 the first V&V standards committee was formed in ASME: Committee on V&V in Computational Solid Mechanics In 2006 the Committee produced ASME V&V : Guide for V&V in Computational Solid Mechanics Codified the concepts of: Conceptual model, mathematical model, and computational model Code verification is distinguished from solution verification Adopted a comprehensive view of validation, i.e., the prediction for the conditions of the application of interest must also satisfy the specified accuracy requirements of the model Uncertainty quantification explicitly required in V&V 8

9 Institute of Electrical and Electronics Engineers IEEE (Trial-Use): Recommended Practice for Distributed Interactive Simulation-Verification, Validation and Accreditation Recommended how-to guidelines for VV&A of distributed interactive simulation exercises (revision of ) IEEE: Standard for Software Verification and Validation Recommended software V&V processes to assess conformance to requirements and intended use of the software (revision of ): IEEE Standard for Software Verification and Validation Software V&V includes management, acquisition, supply, development, operation, and maintenance of software 9

10 Institute of Electrical and Electronics Engineers IEEE: Recommended Practice for VV&A of a Federation an Overlay to the High Level Architecture Federation Development and Execution Process Recommends practices VV&A practices and procedures for a high level federation of software IEEE: Standard for Validation of Computational Electromagnetics Computer Modeling and Simulations Recommends procedures to validate modeling and simulation techniques, codes, and models IEEE: Recommended Practice for Validation of Computational Electromagnetics Computer Modeling and Simulation Shows how to validate solutions using measurements, alternate codes, canonical, or analytical methods 1597 clearly applies to M&S of physical processes 10

11 American Nuclear Society International Organization for Standardization ANS (revision of ): Verification and Validation of Non-Safety-Related Scientific and Engineering Computer Programs for the Nuclear Industry ISO :2006 Part 3: Specification with Guidance for the Validation and Verification of Greenhouse Gas Assertions ISO 14065:2007: Greenhouse Gases Requirements for Greenhouse Gas Validation and Verification Bodies for Use in Accreditation or Other Forms of Recognition ISO 16730:2008: Fire Safety Engineering Assessment, Verification and Validation of Calculation Methods ISO :2010: Industrial Automation Systems and Integration Product Data Representation and Exchange Part 1488: Verification and Validation ISO 14066:2011: Greenhouse Gases Competence Requirements for Greenhouse Gas Validation Teams and Verification Teams 11

12 American Society of Mechanical Engineers American Society of Civil Engineers ASME V&V : Standard for Verification and Validation in Computational Fluid Dynamics and Heat Transfer ASCE Standard 2009: Verification and Validation of 3D Free- Surface Flow Models ASME V&V : An Illustration of the Concepts of Verification and Validation in Computational Solid Mechanics These standards take a M&S V&V perspective 20 conceptually conflicting V&V standards have been produced. Houston, we have a problem. 12

13 Ideas for Resolution of Conflicting Views of V&V What are the primary goals of software V&V vs M&S V&V? Software V&V is focused on software reliability M&S V&V is focused on simulation credibility What does software V&V vs M&S V&V produce? Software V&V: software product M&S V&V: information service 13

14 Four Critical Challenges in M&S V&V: Development of a V&V Plan Example of questions that should be answered in the V&V plan: What is the application domain over which the model is expected to make predictions? What system response quantities (SRQs) is the model expected to predict? What are the code and solution verification requirements? What validation hierarchy is appropriate for the system of interest? What is the validation domain for each tier of the validation hierarchy? What validation metrics are to be used? What are the accuracy requirements for the model in the validation domain? What are the accuracy requirements for the model in the application domain? What are the costs, schedule, and manpower requirements to complete the V&V plan? 14

15 Where Do We Stand with Regard to Constructing and Using V&V Plans? 15

16 Validation Metrics What is a validation metric? A measure of agreement between computational results and experimental measurements for SRQs of interest Steps to evaluate a validation metric result: 1) Choose a system response quantity of interest 2) Experimentally measure, if possible, all input quantities needed for the code 3) Experimentally measure the system response quantity of interest 4) Using the code and all the input data provided, compute the system response quantity of interest 5) Compute a difference between the experimental measurements and the computational results 16

17 Model Accuracy Assessment, Calibration and Prediction (from Oberkampf and Barone, 2006) 17

18 Approaches to Constructing Validation Metrics Hypothesis testing methods Comparing the statistical mean of the simulation and and the mean of the experimental measurements Bayesian methods Assessing if the simulation passes through the scatter in the experimental data Comparison of cumulative distribution functions from the simulation and the experimental measurements (area metric) What are the goals of using a validation metric? Estimating model form uncertainty Assess model adequacy with respect to application requirements 18

19 Extrapolation of Models At each V in the validation domain, one can: Calibrate parameters Compute a validation metric result To approximate the validation metric over the validation domain, one can: Interpolation function Regression fit Beyond the validation domain, one must extrapolate the model (adapted from Trucano et al, 2002) 19

20 Prediction Within the Validation Domain: Interpolation Traditionally, model form uncertainty was not estimated As a result, model form uncertainty was ignored For high-dimensional input spaces, it is difficult to determine if one is interpolating or extrapolating (from Oberkampf and Roy, 2010) 20

21 Prediction Far Outside the Validation Domain: Large Extrapolation Extrapolations can occur in terms of: Input quantities Non-parametric spaces Extrapolation may require: Large changes in coupled physics, e.g., heating effects on structural dynamics Large changes in geometry or subsystem interactions, e.g., partially melted fuel rods in a reactor (from Oberkampf and Roy, 2010) Large extrapolations should result in large increases in uncertainty 21

22 Predictive Capability y = f ( x) x = y = { x 1, x 2, x m } { } y 1, y 2, y n (from Oberkampf and Roy, 2010) 22

23 Sources of Uncertainty Uncertainty in input parameters (mathematical and numerical): Input data parameters (independently measureable and nonmeasureable) Uncertainty modeling parameters Numerical algorithm parameters Numerical solution error: Round-off error Iterative error Spatial and temporal discretization error Model form uncertainty: Estimated over the validation domain using a validation metric Extrapolation of the validation metric outside of the validation domain Estimated at the application conditions using competing models What is included in Predictive Capability? 23

24 Example of Probability-box with a Mixture of Aleatory and Epistemic Uncertainty (from Roy and Oberkampf, 2011) 24

25 Example Showing Total Uncertainty Using Alternate Competing Models Predicted Track of Hurricane Emily 2005 (from Green, 2007) 25

26 Concluding Remarks Conflicting views between software and M&S V&V can be resolved if: It is accepted that we are interested in different deliverables Turf is considered secondary to the advancement of technology and the public good Technical progress is critically needed in: Improved guidance and practice in constructing V&V plans Construction, interpretation, and use of validation metrics Approaches to extrapolating various types of uncertainties Approaches to estimating and interpreting predictive uncertainty Quote from William H. Press: Simulation and mathematical modeling will power the 21 st Century the way steam powered the 19 th. 26

27 References Ayyub, B. M. and G. J. Klir (2006). Uncertainty Modeling and Analysis in Engineering and the Sciences, Boca Raton, FL, Chapman & Hall. Bayarri, M. J., J. O. Berger, R. Paulo, J. Sacks, J. A. Cafeo, J. Cavendish, C. H. Lin, and J. Tu (2007), A Framework for Validation of Computer Models, Technometrics, Vol. 49, No. 2, pp Coleman, H. W. and F. Stern (1997), Uncertainties and CFD Code Validation, Journal of Fluids Engineering, Vol. 119, pp Chen, W., L. Baghdasaryan, T. Buranathiti, and J. Cao (2004), Model Validation via Uncertainty Propagation and Data Transformations, AIAA Journal, Vol. 42, No. 7, pp Chen, W., Y. Xiong, K-L Tsui, and S. Wang (2008), A Design-Driven Validation Approach Using Bayesian Prediction Models, Journal of Mechanical Design, Vol. 130, No. 2. Dowding, K., R. G. Hills, I. Leslie, M. Pilch, B. M. Rutherford, and M. L. Hobbs (2004), Case Study for Model Validation: Assessing a Model for Thermal Decomposition of Polyurethane Foam, Sandia National Laboratories, SAND , Albuquerque, NM. Ferson, S., W. L. Oberkampf, and L. Ginzburg (2008), Model Validation and Predictive Capability for the Thermal Challenge Problem, Computer Methods in Applied Mechanics and Engineering, Vol. 197, pp Ferson, S. and W. L. Oberkampf (2009), Validation of Imprecise Probability Models, International Journal of Reliability and Safety, Vol. 3, No. 1-3, pp

28 References (continued) Ferson, S. and W. T. Tucker (2006), Sensitivity Analysis Using Probability Bounding, Reliability Engineering and System Safety, Vol. 91, No , pp Ferson, S., W. L. Oberkampf, and L. Ginzburg (2008), Model Validation and Predictive Capability for the Thermal Challenge Problem, Computer Methods in Applied Mechanics and Engineering, Vol. 197, pp Green, L. L. (2007), Uncertainty Analysis of Historical Hurricane Data, American Institute of Aeronautics and Astronautics, Paper Helton, J. C., J. D. Johnson, C. J. Sallaberry, and C. B. Storlie (2006), Survey of Sampling- Based Methods for Uncertainty and Sensitivity Analysis, Reliability Engineering and System Safety, Vol. 91, No , pp Hasselman, T. K. (2001), Quantification of Uncertainty in Structural Dynamic Models, Journal of Aerospace Engineering, Vol. 14, No. 4, pp Hills, R. G. (2006), Model Validation: Model Parameter and Measurement Uncertainty, Journal of Heat Transfer, Vol. 128, No. 4, pp Hills, R. G. and T. G. Trucano (2002), "Statistical Validation of Engineering and Scientific Models: A Maximum Likelihood Based Metric," Sandia National Laboratories, SAND , Albuquerque, NM. Kaplan, S. and B. J. Garrick (1981). "On the Quantitative Definition of Risk." Risk Analysis. 1(1),

29 References (continued) Kennedy, M. C. and A. O Hagan (2001), Bayesian Calibration of Computer Models, Journal of the Royal Statistical Society Series B - Statistical Methodology, Vol. 63, No. 3, pp Morgan, M. G. and M. Henrion (1990). Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis. 1st Ed., Cambridge, UK, Cambridge University Press. O Hagan, A. (2006), Bayesian Analysis of Computer Code Outputs: A Tutorial, Reliability Engineering and System Safety, Vol. 91, No , pp Oberkampf, W. L. and M. F. Barone (2006), "Measures of Agreement Between Computation and Experiment: Validation Metrics," Journal of Computational Physics, Vol. 217, No. 1, pp. 5-36; also, Sandia National Laboratories, SAND Oberkampf, W. L. and T. G. Trucano (2002), Verification and Validation in Computational Fluid Dynamics, Progress in Aerospace Sciences, Vol. 38, No. 3, pp Oberkampf, W. L., T. G. Trucano and C. Hirsch (2004). "Verification, Validation, and Predictive Capability in Computational Engineering and Physics." Applied Mechanics Reviews. 57(5), Oberkampf, W. L. and T. G. Trucano (2008). "Verification and Validation Benchmarks." Nuclear Engineering and Design. 238(3), Oberkampf, W.L. and C. J. Roy (2010), Verification and Validation in Scientific Computing, Cambridge University Press, Cambridge, UK. 29

30 References (continued) Roache, P. (2009). Fundamentals of Verification and Validation, Socorro, New Mexico, Hermosa Publishers. Roy, C. J. (2005). "Review of Code and Solution Verification Procedures for Computational Simulation." Journal of Computational Physics. 205(1), Roy, C. J. and W. L. Oberkampf (2011). "A Comprehensive Framework for Verification, Validation, and Uncertainty Quantification in Scientific Computing." Computer Methods in Applied Mechanics and Engineering. 200(25-28), Saltelli, A., M. Ratto, T. Andres, F. Campolongo, J. Cariboni, D. Gatelli, M. Saisana, S. Tarantola (2008), Global Sensitivity Analysis: The Primer, Wiley, Hoboken, NJ. Sprague, M. A. and T. L. Geers (1999), Response of Empty and Fluid-Filled, Submerged Spherical Shells to Plane and Spherical, Step-Exponential Acoustic Waves, Shock and Vibration, Vol. 6, No. 3, pp Stern, F., R. V. Wilson, H. W. Coleman and E. G. Paterson (2001), Comprehensive Approach to Verification and Validation of CFD Simulations-Part 1: Methodology and Procedures, Journal of Fluids Engineering, Vol. 123, No. 4, pp Trucano, T. G., M. Pilch and W. L. Oberkampf. (2002). "General Concepts for Experimental Validation of ASCI Code Applications." Sandia National Laboratories, SAND , Albuquerque, NM. Zhang, R. and S. Mahadevan (2003), Bayesian Methodology for Reliability Model Acceptance, Reliability Engineering and System Safety, Vol. 80, No. 1, pp