Forecasting technologies. Johan Hartnack

Forecasting technologies Johan Hartnack

Forecasting DHI #2

Roles Decision makers Operator/system maintenance Public DHI #3

Needs Overview with quick assessment of operational choices and impacts Configuration and maintenance. Advanced operations of system Clear messages to be understood without specialist knowledge DHI #4

Decision maker Evaluate choices quickly Impact assessment Issue warnings Initiate mitigation measures DHI #5

Operator/system maintenance Model set-up Uncertainty assessment Scenario evaluations Configuration Data hook up DHI #6

Dissemination GIS Time series Temporal development Levels of data access DHI #7

Configuration and operating system Model adaptors configuration through XML files Workflow - Jobs manager Create automatic reports DHI #8

Openness as you need it empowering you DHI #9

Spreadsheets Well known spread sheet functionality Access live data Build models Create reports

Scripting An Iron Python environment API to all modules Functionality to store and manage scripts Scripting Time series GIS Spreadsheet Scenario Jobs Indicator Analysis MCA/CBA Document Metadata DHI

Code your own Tools Develop your own API Access to data Customization

Open Software Architecture DHI

MIKE Operations Desktop Data base WEB GIS Time-series MIKE Scripting Workbench Spreadsheets Jobs Workbench

MIKE Technologies for all Water Environments MIKE MODELS MIKE INFO MIKE PLANNING MIKE OPERATIONS Packaged as standard products Configurable Open (e.g. to data) Extendable (scripting, API)

Aarhus Danish for Progress Aarhus University Architecture and art Recreational and active use of the environment Business and industries DHI #16

Expected project outcome Aarhus Water Aarhus Water Aarhus Water Bathing water in Lake Brabrand (hygienic) Improved water quality/partly bathing water in River Aarhus Bathing water in the Harbour (hygienic) DHI #17

DHI #18

Automated Integrated Modelling MIKE SHE MIKE 11 Data flow Model preparation Model execution Real time control Issues of warning MIKE URBAN MIKE 3 Distributed rainfall from Radar DHI

Real-Time Integrated Control DIMS.CORE Short term rainfall forecast (MAR) Predicted run-off (MIKE URBAN) Dynamic Overflow Risk Analysis (DORA) WWWTP max. hydraulic load Layer 3 Data validation and filtration Software sensors: Flow, Elevations, tank filling, etc. PID (flows) at each storage tank PID (elevations) at each storage tank PID output: 0-100% distributed WISYS to setpoints for pumps, weirs/gates at each storage tank Layer 2 Levels, flows and weir/gate positions Set-points PLC/SCADA Sensors/Actuators Layer 1 DHI

DHI bathing water forecast service DHI #21

Public information on bathing water quality App and web based public warning system DHI #22

One warning system - Integrating data from multiple organizations and authorities Internet Environmental Section Aarhus Municipality Aarhus Water Utility company Waterforecast Operated by DHI 23 DHI

Project Status Bathing water in Lake Brabrand (hygienic) Stiften Politiken Stiften Improved water quality in River Aarhus Bathing water in the Harbour (hygienic) DHI #24

Saving in investment Ordinary and larger retention basins 79 million EUR Controllable and smaller retention basins Automation and control system Total 45,6 million EUR 1,7 million EUR 47,3 million EUR Saving 32 million EUR 40 % DHI #25

Simple and effective web applications for real time monitoring and early warning systems DHI #26

Greve Flood Warning System

Greve Flood Warning System Web solution based on MIKE products MIKE Models Rainfall forecast MIKE21 FM MIKE URBAN CS MIKE OPERATIONS Automatic Operation Results every 4 hours Forecast 24 hours Web solution

Greve Flood Warning System MIKE Web technology Time-series GIS Spreadsheets MIKE Web API Website Polymer/Web Components Configurable MIKE product Project specific implementation

MINERWA - Minimising Non-Revenue Water in distribution networks

LONG-TERM SOLUTION TO SUSTAINABLE WATER NETWORK MANAGEMENT Our solution to Minimise Non-Revenue Water (MINERWA) works with the data you have, follows international recommendations and uses a well-proven methodology and it pays off from the start. Only a minimum of input data is required to establish an overview and understanding. MINERWA offers: a well-structured data repository analytical engines an efficient, yet customisable user-interface with key performance indicators as well as in-depth reporting NRW is produced water, which is not paid for. It can be due to real losses (like leakages) or apparent losses (such as theft or metering inaccuracies). In many parts of the world, water resources are limited and thus NRW can have significant consequences on the level of service as well as on revenue loss DHI

MINERWA benefits reduced water losses reduced pipe burst risks reduced energy consumption Documentation of effects Facilities for for rehabilitation and emergency planning, water quality risk analysis and much more. MINERWA is a solution delivered jointly by DHI and EnviDan International. It is offered as a hosted solution where we take full responsibility of running and maintaining the MINERWA and making it available to you DHI

Minerwa web solution DHI

How to build your own integrated water management system DHI #34

Building blocks an example Data management system Data integration/ validation Processing Reporting SCADA Model analysis Numerical / empirical models of the system Operationalzation Scheduling and coupling of data management system and models Warning Operational people Public Control / optimization Control Optimization Feedback to SCADA Manually Automatic

Our solution builds on generalised software components to provide standard products as well as custom solutions

DHI Software frame work for integrated solutions Integration of data and models MIKE OPERATIONS Data management and integration to SCADA DIMS.CORE Numerical models MIKE URBAN (collection system) MIKE 11 (River) MIKE SHE (Catchment + river) MIKE 3 (Harbour and ocean)

Looking ahead Data Assimilation techniques Faster model execution Hardware Smart systems Surrogate models DHI 38

Data Assimilation to internal measurements DHI #39

Data Assimilation Framework Kalman filter state updating procedure Introduction of boundary errors in each simulation of the ensemble Ensemble statistics Facilitates state updating on a wide range of dependent variables Uncertainty analyses

Input Data assimilation Future Traditional MIKE model Time series Topography Model parameters Uncertainty Uncertainty Uncertainty Measurements Simulation MIKE model Data assimilation Results Main: Water level, Discharge, Velocities, etc. Statistics: Mean values, Covariance, Confidence intervals etc.

Mathematical setting Model description in the form of a model operator x F( x, u k 1 k k ) Where x k The state variables of the system (H-,Q-, depth integrated velocities) F The model operator (a time step in MIKE HYDRO River) k the time step u k The forcing terms of the system (boundary conditions)

Model errors Governing equations Discretization Conceptualization Sub-optimal model parameters Initial conditions Temporal forcing terms Difficult to quantify

Stochastic setting River model as stochastic process e k x F( x, u ε k 1 k k k The model errors of the system ) Updating scheme Measurements included K x Kalman gain matrix Depends on the covariance matrix of state variables Weighting matrix for the system x updated k 1 k 1 KΔ The evaluation of the covariance matrix is the main bottleneck

Ensemble Kalman filter Simulation Ensemble size x 1 M k x 2 k x M k Measurements x 1 k+1 x 2 k+1 x M k+1 Data assimilation Construct Filter x 1 k+1 x 2 k+1 x M k+1 Output Next time step

Model errors Model refinement using an auto regressive process of first order for model errors I.e. Updating can be carried out on the model error

Test example Q t L = 23000 m I = 0.025 % L = 11000 m I = 0.025 % Q/H relation

Effectiveness of updating algorithm Internal point + reference run ooerroneous run D updated

Effectiveness of updating algorithm Downstream boundary internal point + reference run erroneous run D updated

Effectiveness of updating algorithm Upstream boundary condition updating + reference run erroneous run D updated

Uncertainty assessment Method uses an ensemble of simulation thus as an added bonus this ensemble may be used to estimate Confidence intervals (50%, 80 % etc.) Standard deviations (in all points) Valuable tool for sensitivity analysis of boundary conditions

Hardware Brute force for speed DHI #52

Parallelization A case study Christchurch, New Zealand Catchment area approx. 420 km 2 including three river systems in the model domain: Avon River Styx River Heathcote River DHI 2D model domain: 4.2 million elements 10 m x 10 m resolution flexible mesh (rectangular elements) Distributed rainfall-runoff with no losses (rain-on-grid) - 1% AEP event - 21 hour storm

Hybrid Parallelization A case study Christchurch, New Zealand Run time on desktop PC (MPI) is 8.9 hours: 16 core Dell Workstation 2 x Intel Xeon CPU ES- 2687W v2 (8 core, 3.40 GHZ) 32 GB of RAM Windows 7 operating system Run time with 1 x GeForce GTX TITAN GPU card is 3.1 hours Run time with 2 x GeForce GTX TITAN GPU card is 1.7 hours #54 DHI

Simulation time [hours] Hybrid Parallelization Christchurch, New Zealand 1.40 Christchurch MIKE 21 FM model HPC Cluster simulation perfomance 1.20 64; 1.31 1.00 0.80 0.60 128; 0.81 0.40 0.20 0.00 256; 0.45 512; 0.26 768; 0.23 1024; 0.17 Below 10 minutes 0 200 400 600 800 1000 1200 Number of HPC Cores #55 DHI (Simulations executed by HPC Wales)

Klimaspring next generation smart solution DHI

Smart real-time control of water systems (Smart realtidsstyring af vandsystemer) DHI

Klimaspring Scope Scope: Developing an scalable IT-supported system for the real-time monitoring, modelling, warning and management of rainwater in both drainage systems and on the ground. Aim: Reduce the need to invest in enlarging and upgrading the existing drainage system Make managing rainfall less expensive. Open up new possibilities for the use of water on the ground DHI

Danish R&D project financed by RealDania Collaboration project with university, utility and DHI

Improved rainfall forcasting and handling of forcast uncertainties Knowledge of the amount of rainfall provides the basis of improving the system beforehand DHI

Development of computer models for calculation of where and how much rainfall you will have in the system Possibility of calculating the best optimization of the system

Real time control of controllable elements in the sewer system When rainfall starts the system can automatically lead the water to places where it will cause least damage

DHI User surfaces for display of operation and system status Management, planning and operational personnel can see how the system is running or look into control of previous rainfall events

The system will be able to send information to e.g. web/mobile (awaiting final clarification) Information to citizens DHI

Klimaspring system architecture MIKE Powered by DHI software

Key DHI tasks in Klimaspring Model engine optimization Deterministic models Surrogate models Radar data processing Improve data image correction and forecasting Control algoritm optimization Standardaizing and modulazation Integration of data and models Visualisation DHI

Web front-end DHI

Surrogate models Model Predictive Control DHI #68

Why think of a new optimisation approach? Current optimisation system Uses simplified optimisation model (few decision variables) Includes execution of many hundreds of simulations Takes time New optimisation system Uses detailed optimisation model (thousands of decision variables) Model dynamics described by a simplified (surrogate) model Takes few minutes on today s laptop DHI

Local control System-wide Model Predictive Control (MPC)

Operational Workflow Real-time data Data assimilation Forecast data Model prediction Optimisation TOF Implement optimal control until next optimisation Updated real-time and forecast data Model prediction Optimisation Next TOF Time

MPC optimisation framework Physical system HiFi model (MIKE model) Optimiser Mathematical formulation of (simplified) optimisation model Physical system HiFi model (MIKE model) Surrogate model (Linear) Optimiser Mathematical formulation of optimisation model Optimisation/control problem Optimisation/control problem

What is a surrogate model? Derived from the HiFi model (MIKE model) Sufficiently accurate for modelling the most important characteristics relevant for the problem at hand Computationally fast DHI

Real-time Control Framework Optimal combination of HiFi and surrogate models MPC optimisation with surrogate model Implementation of optimal control HiFi model simulation with implemented control and assimilation of system observations Update surrogate model from HiFi model MPC optimisation with surrogate model

Handling uncertainties Uncertainty in model forcing is described by an ensemble forecast MPC model extended to use ensemble model forcing (Multiple MPC) Probability assigned to each ensemble member Will provide optimal control that is robust to forecast uncertainty

The steps 1 MIKE model Surrogate model 2 Formulation of optimisation model Constraints Operation targets Objective function 3 Behind the scenes Automatic setup of MPC optimisation model Efficient optimisation solver

Optimisation of irrigation system CARM, Australia DHI #77

Irrigation Water Delivery Infrastructure DHI 22 June, 2016 #78

CARM operational goals Supply ordered water to the users The right amount at the right time Keep the river in a lean state Minimize losses due to evapotranspiration Leave room for accommodating natural inflow Keep environmental flow requirements at end-of-system Minimise surplus flows at end-of-system DHI

Benefits New technology enables solving large system-wide optimisation and control problems in real-time Problems that we cannot solve today without brute force May be applied within several business areas Reduced flooding Optimized hydropower Environmental protection DHI

Conclusions Forecast system must adapt to roles Openness that suits you Smart water solutions DHI 81

Thank you Johan Hartnack, DHI jnh@dhigroup.com DHI #82