Improvement of model evaluation by incorporating prediction and measurement uncertainty

Transcription

1 Improvement of model evaluation by incorporating prediction and measurement uncertainty Reporter : Li Shuang School of Environment, BNU Date : July 27, International Soil and Water Assessment Tool Conference

2 CONTENTS 01. Background 02. Method and Application 03. Result 04. Main conclusion

3 Part 1 Background

4 1. Background Hydrological models are widely applied in hydrological process modeling. modeling prevention programming Calibration Validation precondition foundation Confidence Model evaluation 2016 International Soil and Water Assessment Tool Conference Li Shuang 4

5 1. Background The evaluation indicators of calibration and validation Mean absolute error (MAE) Root mean square error (RMSE) Index of efficiency (d) 2016 International Soil and Water Assessment Tool Conference Li Shuang 5

6 1. Background operation climate soil human activities Prediction and measurement uncertainty model equation land use instrument system error 2016 International Soil and Water Assessment Tool Conference Li Shuang 6

7 1. Background The evaluation indicators of calibration and validation Mean absolute error (MAE) Root mean square error (RMSE) Index of efficiency (d) 2016 International Soil and Water Assessment Tool Conference Li Shuang 7

8 1. Background operation climate soil Two new approaches land were modeling developed by incorporating use both human uncertainty prediction and measurement activities uncertainty. model equation instrument system error 2016 International Soil and Water Assessment Tool Conference Li Shuang 8

9 Part 2 Method and Application

10 2.1 Method Cumulative Distribution Function Approach (CDFA) Application condition: Measured data and predicted data are continuous or satisfied with certain distribution functions. Monte Carlo Approach (MCA) Application condition: Measured data and predicted data are dispersed. School of Environment, Beijing Normal University Li Shuang 10

11 2.1 Method (1) Cumulative Distribution Function Approach (CDFA) traditional evaluation approach CDFA A topological distance (D) between cumulative distribution functions has been established to replace the traditional error term. School of Environment, Beijing Normal University Li Shuang 11

12 2.1 Method (1) CDFA The flow chart of the process is as following: Done School of Environment, Beijing Normal University Li Shuang 12

13 2.1 Method (1) CDFA Schematic diagram of topological distance D Probability % Probability % topological distance D * Measured data (predicted data) Cumulative distribution curve of prediction uncertainty Cumulative distribution curve of measurement uncertainty Measured data (predicted data) School of Environment, Beijing Normal University Li Shuang 13

14 2.1 Method (2) Monte Carlo Approach (MCA) Ensure the distributions of predicted data and measured data Generate random sample groups evaluation indicators calculation indicators outputting results analysis School of Environment, Beijing Normal University Li Shuang 14

15 2.2 Application Study area: Daning Watershed is located in the central part of the Three Gorges Reservoir Area. Data: Daily stream flow data(2000~2008);monthly sediment data ( ); Monthly total phosphorus (TP) data ( ) School of Environment, Beijing Normal University Li Shuang 15

16 2.2 Application Generation of measurement uncertainty Approach: Root mean square error propagation method(harmel et al., 2009) The probable error range (PER): Best (Ideal) situation Typical situation Worst (Non-ideal) situation Flow Sediment TP non-ideal situation 36% 102% 221% typical situation 9% 16% 26% ideal situation 2% 2% 2% Measurement uncertainty was assumed as normal distribution School of Environment, Beijing Normal University Li Shuang 16

17 2.2 Application Generation of prediction uncertainty SWAT model predicted data SWAT-CUP model Uncertainty interval Model evaluation with CDFA and MCA normal distribution Prediction uncertainty uniform distribution lognormal distribution School of Environment, Beijing Normal University Li Shuang 17

18 Part 3 Result

19 3. Result For CDFA Measurement uncertainty (Normal distribution) Ideal situation Typical situation Prediction uncertainty Normal distribution Uniform distribution Non-ideal situation Lognormal distribution School of Environment, Beijing Normal University Li Shuang 19

20 3. Result For CDFA Measurement uncertainty in ideal situation Table 3.1 For flow evaluation Traditional method CDFA Point-to-point Normal Uniform Lognormal distribution distribution distribution d RMSE MAE Table 3.2 For sediment evaluation Traditional method CDFA Point-to-point Normal Uniform Lognormal distribution distribution distribution d RMSE MAE

21 3. Result For CDFA Measurement uncertainty in ideal situation Table 3.3 For TP evaluation Traditional method CDFA Point-to-point Normal Uniform Lognormal distribution distribution distribution d RMSE MAE School of Environment, Beijing Normal University Li Shuang 21

22 3. Result Typical situation Flow Sediment TP Traditional method CDFA Point-to-point Normal Uniform Lognormal distribution distribution distribution d RMS E MAE d RMS E MAE d RMS E School of Environment, MAE Beijing Normal University Li Shuang

23 3. Result Non-ideal situation Flow Sediment TP Traditional method CDFA Point-to-point Normal Uniform Lognormal distribution distribution distribution d RMS E MAE d RMS E MAE d RMS E School of Environment, MAE Beijing Normal University Li Shuang

24 3. Result For CDFA Normal distribution of prediction uncertainty Table 3.4 For flow evaluation result Traditional method CDFA -- Normal distribution of prediction uncertainty Ideal situation Typical situation Non-ideal situation d RMSE MAE School of Environment, Beijing Normal University Li Shuang 24

25 3. Result For CDFA Table 3.5 For sediment and TP evaluation results Sediment TP Traditional method CDFA -- Normal distribution Point-to-point Ideal situation Typical situation Non-ideal situation d RMSE MAE d RMSE MAE School of Environment, Beijing Normal University Li Shuang 25

26 3. Result For MCA 100 times sampling 5000 times Values of indicators tend to be stable School of Environment, Beijing Normal University Li Shuang 26

27 (Prediction uncertainty).ideal situation.typical situation.non-ideal situation Normal distribution Uniform distribution Lognormal distribution Flow Sediment TP

28 3. Result Table 3.7 Evaluation result of MCA Flow Sediment TP Traditional method MCA -- Prediction uncertainty Point-topoint Normal distribution Uniform distribution Lognormal distribution Ideal situation 0.73~ ~ ~0.74 Typical situation 0.71~ ~ ~ Non-ideal 0.63~ ~ ~0.79 situation Ideal situation 0.610~ ~ ~0.69 Typical situation ~ ~ ~0.68 Non-ideal situation -0.31~ ~ ~0.66 Ideal situation 0.71~ ~ ~0.81 Typical situation 0.62~ ~ ~ Non-ideal -3.10~ ~ ~0.67 situation School of Environment, Beijing Normal University Li Shuang 28

29 Part 4 Main conclusion

30 4. Main conclusion Traditional method doesn t consider the uncertainty. Compared with traditional evaluation method, CDFA and MCA consider the prediction and measurement uncertainty, the evaluation results are more advisable and realistic. The new evaluation methods may be influenced by the measurement and prediction uncertainty. Therefore, in the same situation of measurement uncertainty,evaluation results may be different between different prediction uncertainty distributions; And in the same case of prediction uncertainty distribution, differences exist in the results when measurement uncertainty is different. when the measured error is small and the number of measured data is enough for a specific distribution, the CDFA and MCA methods can be extended to other model applications for a better evaluation. School of Environment, Beijing Normal University Li Shuang 30

31 2016 International Soil and Water Assessment Tool Conference THANKS