Commercial openpdc/openhistorian and Cloud Computing

Transcription

1 Commercial openpdc/openhistorian and Cloud Computing December 9, 2010 Paul Sung President 1

2 Introduction openpdc & openhistorian openpdc pilot project Cloud Storage service Cloud Computing IaaS 2

3 Introduction 3

4 SK Group ( SK group has contributed to the growth and advancement of Korea by strengthening the national key industries including energy and telecommunication. Organization Financial Information (2009) SK Holdings Total Revenue : $79.6 billion Telco. & IT (17.7 %) Energy & Chemicals Telco. & IT Trading & Services Trading & Services (30.8 %) SK Energy SKC SK C&C SK Telecom SK Networks SK Shipping SK E&S SK Gas SK Broadband SK Telink SK E&C SK Securities K-Power SK Chemicals SK Comms. SK Marketing & Company Energy & Chemicals (51.4 %) 4

5 SK C&C ( Total IT service company: system integration, consulting, operations outsourcing, etc. Incorporated CEO Shin Bae, Kim Head Office South Korea Employees 3,300 Operation revenue 1.17 billion USD Operation income 113 million USD Global subsidiaries & branch offices (by the year 2009) 5

6 Phasor, Inc. (PDI) PDI is a US corporation for our synchrophasor business in North America. Provides professional support services to openpdc and openhistorian users PDI s technical staff is fully equipped with a source-code level of knowledge in openpdc and has over 4 years of experiences in the development, implementation and operation of large-scale cloud computing. Open-source based software development Cloud-scale data archival & analytics Cloud computing technology 6

7 Phasor, Inc. (contd.) Partnership with Grid Protection Alliance(GPA) for the Technical service of openpdc/openhistorian open PDC/ openhistorian Users Technical Support Service Phasor, Inc. Technical Cooperation openpdc/ openhistorian Development Grid Protection Alliance PDI and GPA co-develop openhistorian solution which stores/analyzes Cloudscale phasor data 7

8 Phasor, Inc. (contd.) PDI s services for openpdc/openhistorian include: Deployment, implementation and configuration services Software patch and upgrade services Custom software engineering, system integration, trouble-shooting and consulting services Technical support via on-line, , telephone and on-site services Training services Contact Info Address : Deerfield Corporate Center One, Suite 125, Morris Rd. Alpharetta, GA, 30004, US support@phasordata.com Web Site : 8

9 openpdc & openhistorian 9

10 openpdc in Power Grid Antenna Monitoring/Protection Systems State Estimator SCADA/EMS System Other PDC Phasor RTDMS GPS Clock Interface with Action / Output Adapter openpdc Control Systems Input Adapter Action Adapter Output Adapter WAMAC PMU Interface with Input Adapter Power Grid Stabilizer Other System Input 10

11 About openpdc Based on the SuperPDC developed by the TVA(Tennessee Valley Authority), US starting in 2004 Open source project of SuperPDC A complete Phasor Concentrator designed to process streaming timeseries data in real-time Currently SuperPDC in TVA handles (by the end of 2009) Measurement archival rate of 150 million/hr (3.6 billion a day) Space utilization rate of 1.5 GB/hr (36 GB a day) 120 online PMUs 1850 defined measurements 11

12 About openpdc (contd.) Processing and management system for fast and continuous phasor data Supports standard phasor protocols: IEEE C , IEEE , BPA PDCstream, Virginia Tech FNET, SEL Fast Message, Macrodyne, (IEC61850). Built-in concentration engine which sorts the real-time data into frames based on the timestamp associated with each measurement Provides the data quality monitoring functionalities: flat-line test Range test Timestamp test 12

13 Technical diagram of openpdc 13

14 Technical services for openpdc Input adapter configuration connection for power grid input gathering PMU to openpdc connection Action adapter configuration System configuration for special event detection Output adapter configuration System configuration for output data streaming or storage (Local Archiving /or Cloud Storage) System configuration of Hierarchical data gathering architecture 14

15 About openhistorian open source Historian project that is under development by GPA in collaboration with PDI openhistorian is optimized for efficient storage retrieval of large quantities of time-stamped data Processes and analyzes historical phasor data The three major components Archiver: place data into storage Server: make this data easily available Viewer: easily view historian data within any period of time 15

16 Technical services for openhistorian Archiver configuration openpdc to openhistorian connection Other PDC to openhistorian connection development Server configuration storage server management (maintenance and expansion) Interface development with external applications Viewer configuration Customized viewer development Algorithm development analysis /or mining algorithm development & optimization 16

17 OpenHistorian and Hadoop Current issues in Historian High-cost storage (e.g. SAN, NAS) Scalability Reliability openhistorian adopts Hadoop technology Low-cost storage using commodity hardware High scalability using distributed architecture High assurance of data retention - multi copies of original data - self-healing technology 17

18 Hadoop architecture Name (DFS Master) JobTracker (Job Master) ClientAPI TaskTracker (Task Mgmt.) TaskTracker (Task Mgmt.) TaskTracker (Task Mgmt.) (DFS Slave) (DFS Slave) (DFS Slave) : Hadoop MapReduce : Hadoop Distributed File System 18

19 Hadoop Map & Reduce Map function Reduce function Input File A File A-1 File A-2 TaskTracker Map Map Map Task Task Task TaskTracker Partition using key TaskTracker Reduce Task Output File B File B-1 File A-3 File A-N Map Map Map Task Task Task TaskTracker Map Map Map Task Task Task TaskTracker Reduce Task File B-2 META Table Get block list Task assign to each node JobTracker Run on MapReduce framework. Make a Map function & a Reduce funciton. 19 Distributed/Parallel Processing Speed, Scalability

20 OpenPDC pilot project: 20

21 About the openpdc pilot project Sponsored by Korean government for developing a power grid analysis platform based on cloud computing Collaborate with KEPCO KDN(Korea Electric Power Corporation s subsidiary), Korea University and openpdc Community Design/implement a power grid data analysis platform with openpdc and Hadoop (openhistorian) Large-scale & Real-time power grid data gathering and transportation technology Development of Cloud Storage based on HDFS & Distributed DB platform Large-scale Analysis Computing Technology Development based on Hadoop MapReduce 21

22 Technology / Capability Power Grid Platform Power Grid Infra Service Cloud System Power Grid System Power Grid Analysis Platform Generation Transmission Distribution Consumption Transmission System Distribution System Distributed Collector Large volumes of data management and processing Input (PMU, Time series data, etc.) Rule Based Crawler Generic Loader Specific APIs Cloud Storage (Distributed File System) BLK BLK BLK BLK BLK BLK BLK BLK Distributed Parallel Processing Engine Information Analysis Algorithm Set Distributed base System Mart Search Service Statistics Service Reporting Service Analytics Service Personalized Service Issue Management 22

23 Power Grid Analysis Platform Input Distributed Collector Unstructured : PMU data, archived data Semi-Structured : Web/Application log Structured : DB, XML Rule based Crawler Generic data loader Specific APIs Cloud Storage Hadoop Distributed File System (HDFS) Distributed Parallel Processing Engine Hadoop MapReduce Mart Search Service Reporting and Statistics Service Personalized Service 23

24 Power Grid Analysis Platform Gathering Layer PMU PMU openpdc Gathered.d PMU nephee Framework (with OpenPDC) Agent Analysis Layer FTP Mining Hadoop 24

25 Distributed Collector Input Input Collector Cloud Storage Meta handler PMU1 PMU2 PMU3 PMU4 Agent Collector (Time- Series Sorting) Distributed DB PMU5 base PMU6 HBase CloudDB 25

26 Cloud Storage Service: 26

27 Cloud Storage Service EBS(Korean Educational Broadcasting System) was faced to VOD s traffic congestion on March, 2010 Provided commercial cloud storage service in response for that problem within one week using Hadoop DFS and server virtualization The Size of VOD : 20Tbyte Servers The highest traffic volume: 40Gbit / sec Storages Legacy System Cloud Storage 27

28 Cloud Computing IaaS (Infrastructure as a service) 28

29 About Cloud Computing IaaS Provide cloud computing service(like virtualized sever, storage and so on) through dynamic cloud computing infrastructure & Platform based on opensource S/W and commodity H/W There are some benefits Low-cost & high-performance H/W and S/W vendor independency Business agility 29

30 Cloud Computing Architecture Management & Control Service SaaS PaaS IaaS Security Software Monitoring Privacy Metering J2EE 3-tier Web Server Cloud APIs Mining Framework Security WAS RDBMS Framework For Dev. MapReduce Distributed DB Security Billing Cloud Storage Server Virtualization Hardware Network Provisioning SAN/NAS X86 H/W Security Network 30

31 Thank You Please contact PDI for further information at: Paul Sung Tel: Web Site: 31

32 Hadoop Distributed File System File Creation File Name: 09_22.log Size: 128MB 09_22_1.blk 64MB 09_22_2.blk 64MB request create node info complete Name File Meta Info save Rep# 1 Rep# 2 Rep# 3 Rep# 1 Rep# 2 Rep# 3 32

33 Hadoop Distributed File System File Read request file open File Meta Info File Name: 09_22.log Size: 128MB Request block1 node info Name Request block2 read Rep# 1 Rep# 2 09_22_1.blk Rep# 3 Rep# 1 Rep# 2 09_22_2.blk Rep# 3 33

34 Hadoop Distributed File System Failover ( Failure) Retry to other datanode request file open node info Name File Meta Info read Request block1 Send heartbeat message(10 sec) Request replication Find fail node copy Rep# 1 Rep# 2 09_22_1.blk Rep# 3 Rep# Rep# 1 3 Rep# 2 09_22_2.blk Rep# 3 34