Performance Metrics For Evaluating LNG Vapor Dispersion Models

Transcription

1 Performance Metrics For Evaluating LNG Vapor Dispersion Models by Frank A. Licari, PE, CSP United States Department of Transportation Pipeline and Hazardous Materials Safety Administration Pipeline Safety Office Washington, DC Mary Kay O Connor Process Safety Center -1-

2 Agenda Historical Perspective of Metrics Novel Performance Metric - MSWC Methodology to Calculate MSWC Example Calculations Error Analyses & Their Importance Conclusions -2-

3 Traditional Metrics Validate Vapor Dispersion Model Performance Model Comparisons Hanna et al [2] -3-

4 -4- Mary Kay O Connor Process Safety Center Historical Perspective of Metrics 1980s Havens & Spicer DEGADIS 1993 Hanna et al Comparative Study 2001 Carissimo et al SMEDIS Validation 2004 Chang & Hanna Model Performance

5 -5- Mary Kay O Connor Process Safety Center Historical Perspective of Metrics Are Valuable Tools To Validate Dispersion Models Traditional Statistical Methodologies: characterize strengths of models and identify their best applications Yet, Past Metrics Don t Describe: extra separation distance that protects the public or additional confidence in a model s predictions

6 New Metric Describes Model s s Inherent Safety & Confidence -6-

7 Novel Performance Metric - MSWC Margin of Safety With Confidence is a statistical tool quantifies model performance allows models to be compared describes model s minimum margin of safety accurately describes confidence level of model predictions for 30+ data pairs -7-

8 Novel Performance Metric - MSWC Margin of Safety for One Prediction Is Ms i Pi O i Margin of Safety of a Vapor Dispersion Model Is a Range of Values Due to: atmospheric conditions local terrain test error modeling assumptions computational error -8-

9 x Mary Kay O Connor Process Safety Center Methodology to Calculate MSWC LNG Test & Atmospheric Stability Ms i (Pred/ObsRatio) Burro 8 - E DEGADIS Dispersion Model Predictions Burro 9 - C Maplin 29 - D Gas Concentration Data from LNG Field Tests Data Ratios Maplin 39 - D Range =.541 to Table 1 Excerpt of Havens 1992 Gas Concentration Ratios [5] x =

10 Methodology to Calculate MSWC Histogram of Ms i in Table 1-10-

11 Methodology to Calculate MSWC LNG Test & Atmospheric Stability Ms i (Pred/ObsRatio) Burro 8 - E S Ms n i 1 (Ms i n 1 x) 2 =.49 Burro 9 - C Ms desired = 1.0 Maplin 29 - D Z score Msdesired x S Ms = Maplin 39 - D Confidence Level = 72% Table 1 Excerpt of Havens 1992 Gas Concentration Ratios [5] MSWC = 1.0 with 72% Confidence -11-

12 Ms desired Mary Kay O Connor Process Safety Center 72% 1.0 = = 1.28 Figure 1 - Ms desired Z score Determines and Confidence Level -12-

13 Ms desired Mary Kay O Connor Process Safety Center MSWC Explains How Model s Inherent Margin of Safety Is 1.0 or More 72 % of Gas Concentration Predictions Equal or Exceed LNG Field Trial Observations Models May Be Evaluated By Comparing Their inherent margins of safety (or safety buffers) confidence level (bias to over or under predict) -13-

14 MSWC Explains Model s s Accuracy & Shapes Evaluation Decision -14-

15 -15- Mary Kay O Connor Process Safety Center MSWC Example for Distance Predictions Explains Distance Prediction Concepts for Siting LNG Facilities Describes Importance of Societal Risk Preferences Calculates MSWC for 2 Geographic Regions Compares Regional Decisions to Accept DEGADIS Distance Predictions

16 Distance Prediction Concepts Property Line Site property line so hazards of flammable gas during an LNG spill remain in facility 100% LFL Gas concentration at 100 % of the lower flammability limit (LFL) is min. distance 50% LFL Distance to LNG Facility s Property Line 50% LFL is NFPA 59A required distance -16-

17 -17- Mary Kay O Connor Process Safety Center MSWC Example - Societal Risk Preferences Property Line of LNG Facility May Extend to 100% LFL A Ms desired Region A Prefers Safety Buffer & = 1.5 R Ms desired Region R Prefers No Safety Buffer & = 1.0 Each Region Expects DEGADIS Predictions to Have High Confidence Levels

18 MSWC A Example of Distance Predictions LNG Test & Atmospheric Stability Flammability Limit (LFL) Observed Distance (m) Predicted Distance (m) Ms i (Pred/Obs Ratio) Burro 8 - E 50% Burro 9 - C 50% Maplin 29 - D 50% Maplin 39 - D 50% Burro 8 - E 100% 360* 360* 1.000* Burro 9 - C 100% 240* 450* 1.875* Maplin 29 - D 100% 150* 180* 1.200* Maplin 39 - D 100% 125* 220* 1.760* *data extrapolated from Figures 3 through 6 [5] Table A.1 Ms i for Distances at 50 & 100 Percent LFL x A = 1.36 S =.40 = 1.5 =.34 Ms A Ms desired Z score MSWC A = 1.5 with 37% Confidence -18-

19 MSWC R Example of Distance Predictions LNG Test & Atmospheric Stability Flammability Limit (LFL) Observed Distance (m) Predicted Distance (m) Ms i (Pred/Obs Ratio) Burro 8 - E 50% Burro 9 - C 50% Maplin 29 - D 50% Maplin 39 - D 50% Burro 8 - E 100% 360* 360* 1.000* Burro 9 - C 100% 240* 450* 1.875* Maplin 29 - D 100% 150* 180* 1.200* Maplin 39 - D 100% 125* 220* 1.760* *data extrapolated from Figures 3 through 6 [5] Table A.1 Ms i for Distances at 50 & 100 Percent LFL x R = 1.36 S =.40 = 1.0 = -.89 Ms R Ms desired Z score MSWC R = 1.0 with 81% Confidence -19-

20 81% 1.0 = 1.5 of 37% 1.36 Figure 2 - R MSWC MSWC A & Shape an Evaluation Decision -20-

21 Ms desired Mary Kay O Connor Process Safety Center Findings From Distance Prediction Example Siting a LNG Facility Property Line at 100% LFL Reduces Safety Buffer to Zero Desired, Minimum Margin of Safety Shapes Region s Acceptance Decision Region A May Reject Model; It Overpredicts by Factor of 1.5 with 37% Confidence Region R May Accept Model, If Its Constituents Believe Safety Buffer is Unnecessary -21-

22 What Is Size of Model s s Safety Buffer? -22-

23 -23- Ms desired Mary Kay O Connor Process Safety Center MSWC D of Exclusion Zone Predictions Describe Margin of Safety for an Exclusion Zone Prediction Calculate MSWC for Exclusion Zone Predictions Compare MSWC for Exclusion Zone Predictions to Distance Predictions

24 Ms desired Mary Kay O Connor Process Safety Center Exclusion Zone Creates Safety Buffer Property Line 100% LFL 50% LFL During LNG Spill, Exclusion Zone at 50% LFL Separates Public from Hazards of Flammable Gas at 100% LFL -24-

25 Ms desired Mary Kay O Connor Process Safety Center Part 193 Requires Margin of Safety 49CFR Part 193 & NFPA 59A (2001 edition) Establish LNG Facility Property Line at 50% LFL Safety Buffer Protecting Public Is Inherent Margin of Safety of 50 vs. 100% LFL Margin of Safety for DEGADIS Prediction of Exclusion Zone Distance Is: D Msi P O 50% LFL i 100% LFL i -25-

26 MSWC D Example - Exclusion Zone Distances 100% LFL Observed Distance (m)* 50% LFL Predicted Distance (m) Atmospheric (Pred/Obs LNG Tests Stability Ratio) Burro 8 E *data extrapolated from Figures 3 through 6 [5] Ms i Burro 9 C Maplin 29 D Maplin 39 D D Table A.1 D Ms i for Distances at 50 & 100 Percent LFL x A = 2.41 S =.78 = 1.5 = Ms A Ms desired Z score MSWC D = 1.5 with 88% Confidence -26-

27 -27- Ms desired Mary Kay O Connor Process Safety Center MSWC D Compared To Previous Examples Examples MSWC D Prediction Exclusion Zone Distance x Inherent Margin of Safety Confidence Level (%) Safety Buffer robust MSWC R Distance none MSWC A Distance inadequate Societal Preferences Shape Model s Acceptance

28 In MSWC What Is Error s s Confidence Level? -28-

29 -29- Ms desired Mary Kay O Connor Process Safety Center MSWC Error Analyses & Their Importance Sample Sizes in Previous Examples Are Small Calculations Contain Some Statistical Error Ms i Large Datasets with 30+ Minimize Error Error Analyses Characterize MSWC s Accuracy

30 Analysis of E G, Standard Error of Mean LNG Test & Atmospheric Stability G Ms i (Pred/ObsRatio) Burro 8 - E Burro 9 - C xg G Ms desired S Ms = 1.28 G =.49 G Z score = 1.0 = -.57 Maplin 29 - D G t / 2 n G = at 90% = Maplin 39 - D E G t G ( / 2 S Ms n G G ) = Table A.3 Excerpt of Havens 1992 Gas Concentration Ratios [5] min x G or = = 1.10 or 1.46 max G E -30-

31 = of 58% of 83% 1.0 = =.18 = 1.28 Figure A.6 - G MSWC With Min. & Max. Confidence Limits -31-

32 Ms desired Mary Kay O Connor Process Safety Center Confidence Level Accuracy for Small Sample Error in Confidence Level Is Estimated: G E G G G Zs core min or Zscore max Z score -.20 or -.95 S Ms G G Z s G Z score max core min & Respectively Indicate Confidence Levels of 58 & 83% MSWC G Is 1.0 with 72% Confidence with Approximate Errors of -14 and +11% -32-

33 Effective Performance Metrics Ensure Prudent Evaluation Decisions -33-

34 -34- Mary Kay O Connor Process Safety Center Conclusions Uncertainties Must Be Reconciled As Predictions & Observations Are Correlated Model Predictions Vary By Factor of 2 Due to Natural & Stochastic Variability [2] Screening Predictions & Observations Guides Model Validation Process [3]

35 Conclusions Geometric Mean Bias & Geometric Variance Graphs Readily Compare Model Performance [2] Ms i Fractional Results ( % between.5 & 2) Identify Best Applications for Models [4] Larger Validation Datasets Favor New Performance Metrics Like MSWC Together All Metrics Balance Evaluation Decisions -35-

36 Thank You! Questions Frank A. Licari, PE, CSP Phone: (202)

37 -37- Mary Kay O Connor Process Safety Center Backup Slides

38 -38- Mary Kay O Connor Process Safety Center Why Is Confidence Level 37%? Inherent Margin of Safety Is Zero Model s Bias to Overpredict Is Low Hanna Concluded Good Models Are Within Factor of.5 to 2

39 Ms desired Mary Kay O Connor Process Safety Center Confidence Level Accuracy For Large Sample Confidence Level Error Is Estimated By: Ms Z desired Ms Ms score desired or variability in min Ms max Ms Ms Ms S Ms Standard Error of Is: max Ms S Ms Z / 2 1 2n min Ms S Ms Z / 2n 1 2 For 30+ Ms i, Ms Ms & x score variability in Ms Yields Confidence Level Error for Large Sample Z -39-