Ready for the storm. by Paul Nelson. Crown copyright Met Office

Transcription

1 Ready for the storm by Paul Nelson

2 About Paul Nelson Worked at the Met Office for nearly 25 years Observing and support Forecasting for 5 years Large systems Technical Lead, with 20 years of experience in Networking Storage online and offline Virtualization Mainframe Linux and z/os Application development Core intranet services Server/systems consolidation

3 Topics for discussion Overview of the Met Office and infrastructure Data volumes and processing capacity Backup and data management systems Backup and Restore policies Disaster recovery Futures Please feel free to ask questions, these slides can be made available

4 About The Meteorological Office

5 Who do we serve Public weather services, such as the BBC Civil and general aviation Ministry of Defense Science, Research and Space Weather Commercial activities Oil rigs / Supermarkets / Agriculture / Building / Legal Collaboration World Meteorological Organization (WMO) MONSooN Our Collaboration system

6 Met Office locations HQ Met Office, based at Exeter Airports and RAF stations E.g. Belfast / Brize Norton / Lerwick / Marham Regional weather centres E.g. London / Edinburgh / Aberdeen Worldwide locations and offices E.g. Falklands / Gibraltar / Cyprus Everywhere E.g. Mobile Met Unit / Satellites / Ships / Buoys / Road sensors

7 HPC and product delivery in the Met Office Desktop Servers Mainframe Observation sources Archiving System HPC Message Switches and Internet NWP model input data NWP model output data Archivals/Retrievals General purpose access Distributed Operational Systems Global networks

8 Data Volumes and Processing

9 HPC Facts & Figures IBM Power 6 IBM Power 7 Factor Number of Nodes Number of CPUs Total Peak Performance (TFLOPS) Total Memory (TByte) Total Disk (TB) Power (MWatts) MFLOPS/Watt

10 Historic Data growth Jan-00 Jul-00 Jan-01 Jul-01 Jan-02 Jul-02 Jan-03 Jul-03 Jan-04 Jul-04 Jan-05 Jul-05 Jan-06 Jul-06 Jan-07 Jul-07 Jan-08 Jul-08 Jan-09 Jul-09 Jan-10 Jul-10 Jan-11 Jul-11 Jan-12 Jul-12 Jan-13 TB Month

11 Ready for the Storm? 350,000 Data growth for possible HPC options 300, ,000 TB 200, ,000 Option 1 Option 2 Option 3 100,000 50,

12 Approx Met Office backup data 200TB from the HPC supercomputer Mainly user data sent offline to tape 15TB from relational databases Database utilities, mainly keeping second copy online 10TB distributed systems backups User data, program code and configuration files These virtual Linux systems and servers backups go offline 20TB Windows estate Network drive data and Mail exchange sent offline to tape

13 Backup and data management systems

14 Management systems Corporate services RedHat Satellite server for system images & config Several configuration management systems for code and project management and collaboration. Borlands Star team, Git repository / Jira, FCM / Trac Wiki s and Intranet services Mix of database systems and other archives Mail exchange servers Shared file systems for user data Three backups systems and one major archiving system

15 HPC - Backup system Tivoli Storage Manager TSM history / background Tied closely with the HPC service, currently IBM Refreshes every few years with different solutions Quickly restart the Forecast in DR situation 200TB of GPFS file system content backed up /home and /critical 2x500TB file systems 2x225TB file systems mirrored (/critical)

16 HPC - Backup system Tivoli Storage Manager 10TB daily churn of incremental data Increment for ever, 24 hour schedule including housekeeping / reclamation 30 day retention periods Logically partitioned IBM TS3500 tape library 6 x IBM 3592 tape drives (Fibre channel attached) 700 x 1TB 3592 tape media Main challenge - GPFS file system scanning Large number of ensemble objects

17 Backup system - NetBackup Netbackup - history / background Serves the Windows estate Mainly corporate administrative data Licensed by number of media servers, clients, drives etc Semantics Netbackup SAN / volume level backups via NDMP direct to tape 20TB of backups served by a small LTO-4 tape library Mix of retention policies with copies sent offsite Few clients, intended for bulk storage backup

18 Backup systems - Reservoir FDR History / Background Client / Server Mainframe based on a RedHat VM. Shares tape library with the HPC TSM service Licensed per TB Fast dump and restore ( FDR innovation Reservoir ) 8TB of front end disk for fast backup and restore 4 x 3592 tape drives in its own TS3500 library partition 1 TB incremental data per night Vaults to tape at the weekend, essential backups sent offsite. All data compressed at the client and by the tape drives

19 The MASS System Since 2000 the Managed Archive Storage System (MASS) provides storage and restore services for large volumes of Met Office data The largest demand for high volume storage and throughput comes from atmosphere and ocean numerical models run on the HPC, currently 25TB per day average, 40TB per day peak Users restore data to Linux compute servers and their desktop, currently 23TB per day accessed on average, 53TB per day peak Although a long term archive, data is deleted when no longer required Data shared between the Met Office and our collaborators (e.g. NERC)

20 High Performance Storage System - HPSS HPSS Collaboration Hierarchical storage software developed in collaboration with five US Department of Energy Labs since 1992 Source code is available to collaboration members and US stake holders (though HPSS is not open source in the usual sense) Customization is allowed (subject to collaboration review) HPSS, a clustered architecture, unique among tape-based products HPSS scales horizontally almost without limits Horizontal scaling is easy by adding cluster components

21 MASS Architecture Overview Moose HPSS HPC, Server, Desktop Moose client Controller Controller Hot Swap Processor Processor x12 Processor x12 Processor x12 Processor x12 Processor x12 x12 12 x 6 TBytes Disk Put Buffer 10 Gb Ethernet Network Mover Mover x13 Mover x13 Mover x13 Mover x13 Mover x13 Mover x13 x13 Core Server Standby Core Server IBM TS3500 IBM Library TS3500 Plus Robots Library x2 Plus Robots x 2 IBM IBM TS1140 IBM TS1140 x36 TS1140 x36 x2 x36 x2 x IBM 3592 JC media Users and projects with account Oracle Database Meta-data and logging 900 TBytes Disk 8 Gb SAN

22 Tape library systems TS x TS1140 HPSS via Moose clients Moose Reservoir 10 x TS1130 drives 3 x TS IT Halls TSM and Reservoir Use same TS3500 tape library

23 Backup and Restore policies

24 Storage Policies Production Operational and critical user data and systems Resilient RAID-5 SAN based storage Mix of NAS and SCSI attached devices NetApp, IBM and Dell/EMC Data replicated and Mirrored across our IT halls Flash/snap copy made available where required Real time transaction logging for databases Nightly backup system copies/increments changes to tape Critical data and National archives Further replicated, vaulted and sent off-site

25 System re-builds & recovery We always cater for full system re-builds System images and storage configurations are held on central satellite servers Kick start system via http from initial RAM disks Use LVM to scan for user/application storage Use SAN based replication to recover storage If user/application storage unavailable, re-apply storage configuration scripts from the RedHat Satellite Restore data from Reservoir, third party backup system Re-start service

26 How we view Backup or Archive? Backup data typically short retention periods Reconstitute/re-build key systems Duplicate copies made for offsite Backup collocation jobs for timely restore Policies defined by system administrators and data owners Archive data typically retained for years Bulk data (model experiments) kept for long periods Largest file approx 500GB compressed; 500MB average Deletion manual process, managed by user, GUI toolset Data duplicated but NOT sent offsite

27 Disaster recovery

28 Disaster recovery on site RAID failures, single system or Storage Head failures We rely on dual hall resilience and data replication Virtual guests can be dynamically moved Application clustering across IT halls and sites Automation and monitoring Human error or SAN corruption. Human error most likely risk Mirroring technologies will NOT save you It s back to snap/flash copies if your lucky It s back to your tape system if your unlucky

29 Site down Nationally important archive data is sent offsite Washington takes over our data feeds for key customers Mainly aviation and MOD European Met Service (ECMWF), located in the UK, run their own models and forecasts Met Office locations around the country can serve the public/media using European and Washington data

30 Futures What next?

31 Futures Offsite facilities and replication being considered Likely to be a collaboration De-duplication and segmentation both on the SAN and backup system, where appropriate Content management systems for improving data management and visibility Reduction in backup data; Removal of the HPC backup service Increasing staff count for data management

32 Collaboration Big part of our corporate strategy NERC National Environment Research Council NERC is the UK's main agency for funding and managing research, training and knowledge exchange in the environmental sciences Access Met Office HPC and Mass / HPSS system via the SuperJANET link (100Gbit backbone) MONSooN - is an acronym (of sorts) for the Met Office (and) NERC Supercomputing (oo) Node 13% of the new IBM in the Met Office

33 Questions? Contact details: Phone: