SeqFEWS: A data-centric workflow manager in the era of Monte-Carlo

Transcription

1 SeqFEWS: A data-centric workflow manager in the era of Monte-Carlo 2018 AUS FEWS Users Conference Lindsay Millard, Hydrologist

2 8 AEP x 16 Durations x 10 Temporal Patterns x 5 scenarios x 4 locations x 30 stochastic samples x Several model revisions =?

3 Presentation Outline Context Seqwater s Dams Quick tour of previous work Building blocks in FEWS, and How to apply it to design workflows Key take-aways

4 Acknowledgements Terry Malone Michel Raymond David Pokarier/Steve Wang Deltares & Seqwater staff

5 Seqwater s Dams 26 referable dams 4 regulated water supply dams 22 unregulated water supply dams Catchment areas range from 10 to 7,000km 2

6 Question What is a Dutch word to describe FEWS? Datafabriek Data-centric Workflow Manager Why not use it for non-forecasting purposes? It would: assist with task efficiency and maximising skillsets avoid buying and learning new software enable efficient data management of AR&R 2016 centralise and allow auditing of tailored design engineering workflows

7 Motivation Wrapping together requirements of AR&R 2016 workflows Keeping workflows efficient and archived python scripts, GIS extraction, etc Scenario management Auditing and Continual Improvement Data sharing/record of project work Feed forward into next flood event or project Gen.Adapter Import Transformations Export

8 Hydrological Cycle & FEWS Closing the loop: Forecast Engineering Real-Time Hydrology Event preparedness Event Calibration Design Engineering Upgrade scenario models Design Hydrology & Hydraulic s

9 Design Hydrology: 21 on-stream Dams: 7 dams > 100km 2 catchment 30km 2 < 12 dams < 100km 2 2 dams <30km 2 Critical Duration is short (<24h) but not short enough (>6h)

10 Catalogue of Catchment Average Rainfall:

11 Project Summary: Improved understanding of temporal patterns Created a catalogue for different duration and catchment sizes Complete analysis: Need for a suite of moderate (6 24h) duration temporal patterns to allow improved distinction between PMF and PMP-DF. Operationalise the data: Make the information readily available to Seqwater staff

12 Related work Examples of completed Standalones GateOPS Next Gen RTC tools module AWAP/Silo dataset Storage of daily rain grids from 1900 Somerset Physical Model Transducers to Animation Stochastic Storm event database 60 Synthetic storms AWRA-L Initial Loss modelling Monte Carlo Hydrology/Hydraulic Interface Treasury Next FEW slides used with Permission of Authors

13 Stochastic Storm Database WaterCoach Synthetic rainfall events were: Re-formatted and imported into FEWS Exported in NetCDF-Grid format

14 AWRA-L Analysis Seq-FEWS Implementation Configure best fit equation parameters Develop grid and scalar displays Integrate initial loss estimates into existing reports* Finalise report and procedures Operational system for Initial Loss estimate

15 Brisbane River Catchment Flood Study Tb of data to harvest 15

16 FEWS building blocks: An application that manages model runs efficiently Management of model queue to: assist event and scenario runs maximise license/hardware utilisation Import/Export Timeseries: Point, Grid and export self-contained NC. Transformations of Timeseries: Grid-grid interpolation / 2D Lookup Tables

17 AR&R 2016 Design Event Workflow Model Build Calibrate Events Define URBS Parameters Storm Hyetographs Download Scrape AR&R Datahub Chop up BoM IFD Grids Preburst Initial /Continuing Loss estimation Design Ensemble Process: Areal Reduction Factors Temporal patterns Design Scenarios Post process Results Statistical Analysis & Boxplots Critical Duration Best Estimate for each AEP Plot and Report Results (1,000s)

18 SeqFEWS and Python Link websites/models using adaptors URBS MIKE11 Tuflow is available in natively HEC RAS 4.1 adapter [roll your own] MIKE 1D/Flood requires Iron Python HEC RAS 5.0.4?? GoldSIM / RORB Machine Learning Matplotlib/Pandas/Seaborn Boxplots! Import Translate Manipulation Storage Export Publish to Reports.py Use General Adapter to execute script.py or batch 18

19 SeqFEWS 0d, 1d, 2d T.S. transformation \ModuleConfig\ \RegionConfig\ \WorkflowFiles\ General Adapter Plots export Module Data set import.xml Runtime parameters \ModuleDataset\ model.zip.xml.nc.csv.bat /.py.bat /.py Model Input \Modules\model\ Model Results/Log Run files Geometry Input TS Model.exe Output TS Grid

20 Export Rainfall from FEWS datastore URBS input Import Q h into FEWS datastore URBS output Q h TS Compute max Q and h from FEWS datastore Max Q h Importance sampling TPT process AEP Calculate for Q & h Store AEP Q and h in FEWS datastore Slice for location AEP frequency

21 Work in progress

22 Key Takeaways Variety of different types of models available in FEWS All stitched together using the adapter concept Python is a toolbox to overcome non-standard issues Models can be mixed in a single workflow for auditing Increasing use of distributed & complex models in workflows Issues: speed, database sizes, complexity, Keep the design workflow organised and repeatable 22

23 Tenets of modelling All Models are wrong Models are never finished, only abandoned If you can t make it perfect, make it adjustable

24 24 Temp/Spatial Patterns Scenario Losses IFDs Global Input Data Rain Losses Event Loop Volume URBS GateOPS.py Peak Q, H, Vol TPT Monte Carlo.py NetCDF TUFLOW Estry.py

25 25 Monte-Carlo Sampling Workflow 1 Select a rainfall burst duration (6 different durations) 2 Select an AEP and the corresponding av. catchment rainfall 3 Sample a space-time pattern of rainfall that satisfies constraints 6 Repeat Steps 3 to 5 for 21 simulations at the selected AEP 5 Run URBS model & Dam Operations model with other (fixed) design inputs 4 Sample initial loss value initial reservoir volume 7 Repeat Steps 2 to 6 for 60 AEPs at selected duration 8 Repeat Steps 1 to 7 for 6 durations 21 x 60 x 6 = Analyse results of simulations (TPT Analysis)

26