Model Diagnostics. How to evaluate and improve a model. CIVE 781: Principles of Hydrologic Modelling University of Waterloo Jun 19 24, 2017

Transcription

1 Model Diagnostics: How to evaluate and improve a model CIVE 781: Principles of Hydrologic Modelling University of Waterloo Jun 19 24, 2017 Model Diagnostics Budget evaluation Hydrograph evaluation How to interpret modeled hydrograph deficiencies Model diagnostics Nash Sutcliffe, Pct. Bias, and the more exotic Strengths and benefits Other data snowpack, ET, etc. Assessing partitioning 2 1

2 Model Diagnostics: First Step Before even picking up a model, look at the data!: Precip & Outflow Snow storage, if available Everything you have! Read about similar landscapes Default Analysis: Monthly & Annual water balance Bulk runoff coefficients Baseflow separation Understand the watershed before youe try to understand the model of the watershed 3 Basics: Monthly Water Balance Monthly ET estimates can be using something as simple as Hargreaves or Thornthwaite Penman often overkill at monthly wshed scale 4 2

3 Thornthwaite Water Budget Diagram From Ekinci et al., Shallow and deep seated regolith slides on deforested slopes in Çanakkale, NW Turkey, Geomorphology 201:70 79 November Thornthwaite Water Budget Very simple emprical monthly model Useful order of magnitude assesment of water budget for your area of interest More value in averaging out over multiple years 6 3

4 Annual Budget Most systems are quasi steady (Δ 0 at the annual scale Δ Δ 0 By estimating baseflow, we can get reasonable ranges of annual water budget partitioning to guide conceptual model/assess model Very simple to calculate Comparison of estimates will help evaluate quality of closure Notes: Disregards interannual storage Depends upon quality of Baseflow estimate Ignores GW includes sublimation 7 Annual Budget: L vovich Partitioning Some of these L vovich relations are surprisingly well defined from year to year Misfit of model from these data clouds can indicate improper representation of processes Simple examination of relations gets us reasonable ranges of annual behaviour Ref.: Sivapalan, et al. (2011), Functional model of water balance variability at the catchment scale: 1. Evidence of hydrologic similarity and space time symmetry, Water Resour. Res., 47(2). 8 4

5 Manual Calibration 9 Typical (Eyes closed) Analysis Automatically calibrate to RMSE No understanding is provided regarding flow partitioning in the system and/or water balance components (e.g., evapotranspiration). a curve fitting exercise DON T DO (ONLY) THIS 10 5

6 The Looong hydrograph 11 Value of Manual Calibration Get direct feel for parameter influence Visualize impact on hydrograph Better feel for parameter compensation Human brain is better than NSE at seeing patterns 13 6

7 Manual Calibration Useful strategy Get volume right (cumulative outflow=cumulative observed outflow) Rainfall/snowfall bias ET parameterization Look for accumulating storage Get freshet right Focus on snow parameters Get recession/baseflow right Focus on soil parameters Resolve individual events 14 Manual Calibration: Large Watersheds Calibrate from upstream to downstream Force data (override streamflow) during calibration Useful window into compensation Enlightening Likely to reduce overall quality before recalibration Seasonal calibration Be wary of winter 15 7

8 Daily/sub daily analysis: Interpreting a Hydrograph Different parts of the hydrograph are controlled by different types of parameters When looking at modeled vs. observed hydrographs, we can sometimes trace the symptoms back to the parameters causing them Non linearity makes this non trivial, but the exercise is worthwhile Useful to understand how the model might be cheating to fit the hydrograph and what this could mean 16 HydrographDecomposition While calibrating/assessing model it helps to mentally decompose the hydrograph into: Freshet events Rainfall events on dry soil Rainfall events on wet/frozen soil Interflow Baseflow Rising limb/falling limb These each have unique controlling parameters Useful for understanding and for manual calibration 17 8

9 Decomposition Snowmelt Rainfall ET Available water Topsoil Abstraction Outflow Lower soil groundwater 18 Hint One useful thing about Raven (and sometimes possible with other models) is that we can typically turn off or mark baseflow/interflow and directly compare the parts of the hydrograph we are interested in 19 9

10 Baseflow Controlled by (1) how much water is infiltrating (2) how much groundwater/lake storage is available Controlled in part by infiltration parameters Modelled baseflow not equivalent to separated baseflow Still useful R packages for baseflow separation: Flowscreen bf_eckhardt() Eckhardt, K Technical note: Analytical sensitivity analysis of two parameter recursive digital baseflow separation filter. Hydrology and Earth System Sciences 16: EcohydRology BaseflowSeparation() Nathan, R. J. and T. A. McMahon (1990). Evaluation of automated techniques for baseflow and recession analysis. Water Resources Research 26(7): Hydrostats baseflows() Ladson, A. R., R. Brown, B. Neal and R. Nathan (2013) A standard approach to baseflow separation using the Lyne and Hollick filter. Australian Journal of Water Resources 17(1): Transport in Raven Raven supports advective transport simulation Track conservative tracers, isotopes, contaminants, nutrients, baseflow, quickflow, snowmelt In HRUs In channel Provides concentrations and fluxes Independent of hydrologic conceptual model; sits atop any model configuration 21 10

11 In.rvi file: Transport: Raven Syntax # # Transport calculations for estimation of snowmelt and glacier # contributions to runoff # Constituent name :Transport SNOWMELT :FixedConcentration SNOWMELT SNOW 1.0 :FixedConcentration SNOWMELT SHALLOW_GW 0.0 :FixedConcentration SNOWMELT DEEP_GW 0.0 :Transport GLACIER :FixedConcentration GLACIER GLACIER 1.0 :FixedConcentration GLACIER SHALLOW_GW 0.0 :FixedConcentration GLACIER DEEP_GW 0.0 We generally only care about snowmelt in quickflow Creates new output files: Runname_concentrations.csv (concentrations by storage unit across the watershed vs. time) Runname_pollutographs.csv (outflow concentration vs. time) 22 Baseflow Transport Can easily be used to track source of flows For example, Track baseflow tag all water leaving groundwater with C=1 [volumetric concentration: vol. GW/vol. water] :Transport BASEFLOW :FixedConcentration BASEFLOW SOIL[3] 1.0 Mass flux = = fraction sourced from groundwater (baseflow) A model based baseflow separation method Can similarly generate (e.g.) volumetric flows sourced from snowmelt/glaciers Useful diagnostic information % of flows sourced from different processes/storage source 23 11

12 Examples baseflow Snow sourced 24 Troubleshooting Common problems arise in hydrograph fitting Instead of randomly turning knobs, it is useful to think about physical nature of problem What processes might control this? What parameters control those processes? Model specific 25 12

13 Symptom: Troubleshooting Dry season hydrograph events are too large in magnitude Potential causes Soil unable to get dry enough underestimates infiltration capacity Modelled ET not high enough Controlled by soil evaporation parameters, potentially crop properties 26 Troubleshooting Symptom: Underestimated freshet, spurious winter flow events Potential causes: Overestimating potential melt during winter 27 13

14 Other things to check Quasi steady soil moisture, depression storage Should exhibit roughly annual cycles in most landscapes with wet and dry years, but some form of equilibrium If water is accumulating in one store over time, it is an indication your model is missing or misrepresenting a process Not letting the store drain enough or not supplying enough water 28 Standard model evaluation metrics How do we quantify goodnes of fit? RMSE NSE Pct. Bias Peak errors Other hydrologic signatures Focus on what they do and do not tell us Some thoughts on metrics for calibration 29 14

15 RMSE A standard error measure for any data: 1 where is the number of observations, and are the i th modeled and observed flows, respectively What it tells us: Everything if zero, not much otherwise Somewhat useful for comparing a model to itself Absolute magnitude is useless (depends on flow magnitude) 30 Nash Sutcliffe Nash Sutcliffe Efficiency (Nash, 1970): 1 Normalizes RMSE in such a way as to support comparison between different models 1 for perfection, 0 for model worse than the most naïve guess (that = ) Critical (and portable) concept: Compares model to most naïve possible model 31 15

16 32 Nash Sutcliffe NSE=0.59 NSE=

17 Beware the Charlatans! 35 Nash Sutcliffe Errors in mean flow We need to develop metrics that are robust with respect to diagnostics related to system s behavior. Ref.: Martinez and Gupta (2010), Toward improved identification of hydrological models: A diagnostic evaluation of the "abcd" monthly water balance model for the conterminous United States, Water Resour. Res., 46. Errors in variance of flow 36 17

18 Percent Bias An indicator of the under/overestimation of total volume released from a basin 37 Peak Errors Peaks are typically most challeninging to predictably resolve Also the target of most applications Highest degree of uncertainty in both data and model Hourly peak daily peak Can generally fix consistent biases (under/overprediction) Forecaster (human) skill takes care of rest 38 18

19 Additional Diagnostics Annual water balance, e.g., the Budyko curve PET/P < 1 Energy limited PET/P > 1 Water limited Frequency representation, e.g., Flow Duration Curve 39 Additional Diagnostics Using soil moisture data to identify model structure At Field Capacity Ref.: McMillan, et al. (2011), Hydrological field data from a modeller s perspective: Part 1. Diagnostic tests for model structure, Hydrological Processes, 25,

20 The value of relevant signatures Functional approach to flow partitioning Resulting relationships between processes: Ref.: Sivapalan, et al. (2011), Functional model of water balance variability at the catchment scale: 1. Evidence of hydrologic similarity and space time symmetry, Water Resour. Res., 47(2). 41 Raven Diagnostics Can use any observation time series Irregular or regularly spaced observations Laundry list of diagnostic measures: NASH_SUTCLIFFE, RMSE, KLING_GUPTA, PCT_BIAS ABSERR, ABSMAX, PDIFF, TMVOL, RCOEF NSC, RSR, R2, CUMUL_FLOW, LOG_NASH,, NASH_SUTCLIFFE_DER, RMSE_DER, KLING_GUPTA_DER 42 20

21 What most metrics can t capture Timing Issues Since most metrics (NSE, RMSE, etc.) are point wise comparisons, they can miss perfect fits This is a serious issue for automated calibration, where we guide model parameter tweaking with these metrics 43 The Value of Alternative data ANYTHING in addition to an observed hydrograph is incredibly helpful Constrains partitioning and proper storage Helps get right results for right reason Input Data (observed forcings) SW radiation Target data (observations of state variables/fluxes) Snow depth Reservoir stage Eddy covariance station AET Isotopes (timing) 44 21

22 Modelling without a Hydrograph? Challenging even with volumes of other information: missing the integrated response Extrapolating such a model to flow problematic 46 Parameter control The knobs Some exclusively control magnitude Some exclusively control partitioning Some exclusively control timing Some do both Partitioning to snow 47 22

23 Decomposition Snowmelt Rainfall ET Available water Topsoil Abstraction Outflow Lower soil groundwater 48 The Model Evolution Process A good model takes time and care (and love) and patience The best operational forecasting models have been massaged and tweaked over many years (and many $$s) They still require human guidance to be effective Very hard to just throw together working model of a basin Lots of attempts to do this. Close (or not so close) inspections of the literature indicate we are not as good at this as we pretend 49 23

24 Model Development From HSPF overview, Aqua Terra/USEPA 50 Take Home Manually calibrate first!! Use more than one metric NSE, Pct bias useful for communicating results Details matter Metric conditional upon model purpose Use all available data to assess and train model Focus on what the model is doing wrong not just on what you are getting right Necessary for model improvement Getting a model wrong is where the most learning happens 51 24