Research on the Framework and Data Fusion of an Energy Big-data Platform

Size: px

Start display at page:

Download "Research on the Framework and Data Fusion of an Energy Big-data Platform"

Leslie Ross
5 years ago
Views:

1 1 Paper Number: 17PESGM2652 Panel: Big data for Integrated Energy Systems Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong Bie, Jiang Wu, Cheng Li Xi an Jiaotong University 21 July 2017

2 2 Content 01 Integrated Energy System 02 Framework of an Energy Big-data Platform 03 Multi-source Heterogeneous Data Fusion 04 Conclusions and Future Work

3 3 Integrated Energy System (Energy Internet)

4 01 Integrated Energy System New challenges Sustainable development Energy crisis Environmental degradation Global climate change Revolution of energy

Revolution: Nuclear power, computers 4 st Industrial Revolution: Cyberphysical system Time: end of 18 th century start of 20 th century 70s of 20 th century

4 4 01 Integrated Energy System New challenges Sustainable development Energy crisis Environmental degradation Global climate change Revolution of energy production and consumption 1 st Industrial Revolution: Appearance of steam engine 2 st Industrial Revolution: Wide use of electricity 3 st Industrial Revolution: Nuclear power, computers 4 st Industrial Revolution: Cyberphysical system Time: end of 18 th century start of 20 th century 70s of 20 th century now Integrated Energy System This new concept attracts a lot of attention Highly integrated and interdependent energy and cyber systems There is no consensus on an exact definition

5 01 Integrated Energy System Integrated Energy System - seen from power system field Centralized resources renewable energy Demand response renewable energy Power system at core

5 5 01 Integrated Energy System Integrated Energy System - seen from power system field Centralized resources renewable energy Demand response renewable energy Power system at core Maximum integration of renewable energy Centralized and distributed resources Integrated energy system considering demand response A new generation of energy system with electricity at core

6 01 Integrated Energy System 6 Integrated Energy System - seen from cyber system field To innovate the current energy system based on the benefits of the cyber system, including openness, free flow of energy and peer-access

build a Global Integrated Energy System, based on ultra

7 01 Integrated Energy System Integrated Energy System - seen from industry field 7 Global Integrated Energy System State Grid: to build a Global Integrated Energy System, based on ultra high-voltage electric transmission system, sharing and utilization renewable energy

8 01 Integrated Energy System Integrated Energy System - seen from finance field 8 cloud platform Business mode of Integrated Energy System Based on new technology and development trend Create revenue for customers and service providers Innovation for energy supply 8

9 01 Integrated Energy System Electricity field 9 Automation field Industry field Integrated Energy System Finance field Cyber field

10 10 01 Integrated Energy System 2016 National Key Research and Development Program of China Researches on basic theory of planning, operation and trading for Integrated Energy System (2016YFB ) (Principal Investigator)

11 01 Integrated Energy System 11 Wind Farm PV Power Station Traditional power system Tower Tower Transmission Line Tower Substation Substation Resident Users Power Plant Factory generation transmission distribution Residential Building Tower Business Building utilization

12 01 Integrated Energy System Architecture Wind Farm Tower Tower Charging Pile

Product H 2 Substation Substation Resident Users Power Plant Factory Product CH 4

Building Tower Store H 2 H 2 Storage Container Product CH 4 Natural Gas Pipeline

Heat Storage Business Building Electrified Traffic System Information Exchange

Equipment 1. A variety of primary energy including wind, solar, gas and coal; 2.

12 12 01 Integrated Energy System Architecture Wind Farm Tower Tower Charging Pile Hydrogen Production Plant Product H 2 PV Power Station Tower Transmission Line EV Product H 2 Substation Substation Resident Users Power Plant Factory Product CH 4 Generate Electricity Using Natural Gas Gas Power Plant Gas Power Plant Residential Building Tower Store H 2 H 2 Storage Container Product CH 4 Natural Gas Pipeline Pressurizer Natural Gas Pipeline Information Exchange Pressurizer Heating Station Heat Storage Business Building Electrified Traffic System Information Exchange Source Information Exchange the Internet Information Exchange Cloud Computing Equipment 1. A variety of primary energy including wind, solar, gas and coal; 2. Spatial and temporal distribution analysis of different resources; 3. Intermittent of renewable energy;

13 13 01 Integrated Energy System Architecture Wind Farm Tower Tower Charging Pile Hydrogen Production Plant Product H 2 PV Power Station Tower Transmission Line EV Product H 2 Substation Substation Resident Users Power Plant Factory Product CH 4 Generate Electricity Using Natural Gas Gas Power Plant Gas Power Plant Residential Building Tower Store H 2 H 2 Storage Container Product CH 4 Natural Gas Pipeline Pressurizer Natural Gas Pipeline Information Exchange Pressurizer Heating Station Heat Storage Business Building Electrified Traffic System Information Exchange Source Network Information Exchange the Internet Information Exchange Cloud Computing Equipment 1. Coupled Natural Gas and Electric Power Network; 2. Large scale, different time scale flows: electricity and gas flow; 3. Big data from both Natural Gas and Electric Power Systems.

14 14 01 Integrated Energy System Architecture Wind Farm Tower Tower Charging Pile Hydrogen Production Plant Product H 2 PV Power Station Tower Transmission Line EV Product H 2 Substation Substation Resident Users Power Plant Factory Product CH 4 Generate Electricity Using Natural Gas Gas Power Plant Gas Power Plant Residential Building Tower Store H 2 H 2 Storage Container Product CH 4 Natural Gas Pipeline Pressurizer Natural Gas Pipeline Information Exchange Pressurizer Heating Station Heat Storage Business Building Electrified Traffic System Information Exchange the Internet Information Exchange Source Network Demand Information Exchange Cloud Computing Equipment 1. Distributed sources: CHP, DG, etc.; 2. Coupled with Transportation Systems via electric vehicles; 3. Multi-energy demand and transform: cooling, heating, gas, electricity.

15 01 Integrated Energy System 15 Modeling Modeling of of cyber-physical Integrated Energy system Systems based on Energy Internet Mixed dynamic system modeing United Interface system modeling Stochastic system simulation and evaluation stochastic modeling of compatibility modeling System planning Planning and business Trading model for Mechanism Energy Internet Coordinated planning Integrated market Distributed game theory analysis Optimal Optimal operation operation and and dispatch dispatch for Energy Internet Stochastic optimization Integrated risk analysis Distributed optimization algorithm Planning platform Trading simulation platform Big data center Operation simulation platform simulation Comprehensive Comprehensive demonstration demonstration platform of system Integrated of Energy Systems Internet demonstration

16 16 Framework of an Energy Big-data Platform

17 02 Framework of an Energy Big-data Platform Data for demonstration Jiuquan wind power base 17 Hydropower base of the Yellow River Electric Power Network of Shaanxi high energy-consuming enterprises Electric Power Network of Northwest China Energy base of northern Shaanxi Distributed multiple energy system Energy system of Xi an

18 02 Framework of an Energy Big-data Platform 18 Features of the Energy Big-data Platform Planning Platform Trading Platform Collection of multi-type energy data Big-data Platform Fast Query of multi-type energy data Data processing and computing Operation Platform

19 02 Framework of an Energy Big-data Platform 19 (1) Collection of multi-type energy data 1 Real-time Data Collection Wind Farm PV Station H 2 Production Plant... Kafka Cluster Hadoop Cluster Real-time Monitoring Database Real-time data flow collection based on Kafka

Challenge Sources of data various Heterogeneous characteristics Solution Multi-source Heterogeneous

20 02 Framework of an Energy Big-data Platform 20 (1) Collection of multi-type energy data 2 Storage 3 Data Fusion Challenge Big data volume(tb level) Solution Hadoop Distributed File System (HDFS) Challenge Sources of data various Heterogeneous characteristics Solution Multi-source Heterogeneous Data Fusion Raw Data Semantic heterogeneous data fusion & System heterogeneous data fusion Big-data Platform

02 Framework of an Energy Big-data Platform 21 (2) Fast Query of multi-type energy data Challenge Big data rapid indexing Solution A distributed, scalable, big data store:

21 02 Framework of an Energy Big-data Platform 21 (2) Fast Query of multi-type energy data Challenge Big data rapid indexing Solution A distributed, scalable, big data store: Apache Hbase. Hadoop Data Dase Hbase Basic storage support HDFS Computing power support MapReduce Coordination services & Failover ZooKeeper High level language support Hive

22 02 Framework of an Energy Big-data Platform 22 (3) Data processing and computing Mining Optimization Visualization Load Forecasting Planning Optimization Planning Platform Operation Platform Report Form State Assessment Power Quality Monitoring Operation Optimization Market Optimization Trading Platform Summary Graph User Interface

23 02 Framework of an Energy Big-data Platform Theoretical Architecture 03 Framework Physical Architecture 02 Technology Architecture 04 Core Technology and Function Big-data Platform

24 Communication Network Interactive interface 02 Framework of an Energy Big-data Platform 24 (1) Theoretical Architecture Data Source Big-data Platform Power system data Natural gas system data Heating system data Meteorological data Data Storage Rapid Index Offline Computing Online Computing Optimization Planning Platform Operation Platform Trading Platform Financial market data Resource Management

25 02 Framework of an Energy Big-data Platform 25 (2) Technology Architecture User Application Layer User Gateway Layer Base Platform Layer Infrastructure Layer Application Gateway Base Platform Infrastructure Surveillance System Offline Computing Online Computing Optimization Spark Client Hive Client Spark Wind Power Data Simulation Yarn HDFS Mapreduce PV Data Simulation Hardware Resources Hadoop Client Storm Client Kafka Storm Building Users Data Simulation Universal Optimiz- -ation Platform Standalone Hive Factory Users Data Simulation Operating Environment Server #1 Server #n Redhat Linux 6.2 Resource Management Universal Optimization Platform Client Distributed Collaborative Framework ZOO KEEPER ( )

26 02 Framework of an Energy Big-data Platform 26 (3) Physical Architecture The layers of the Big-data platform are deployed to the physical nodes, which are connected as a whole through the LAN, providing physical support for applications.

27 27 02 Framework of an Energy Big-data Platform (4) Core Technology and Function Data Processing and Computing Surveillance System Offline Computing Realtime Computing Optimization Random Characteristic Analysis Spark Mllib Mapreduce Resource and Job Management Yarn HDFS Realtime Analysis Spark Streaming Storm Data Storage Data Collection Kafka Standalone Hive Distributed Optimization Universal Optimiz- -ation Platform Distributed Collaborative Framework ZOO KEEPER ( ) Wind Power Data Simulation PV Data Simulation Building Users Data Simulation Factory Users Data Simulation Hardware Resources Server #1 Server #2 Server #n

28 28 Multi-source Heterogeneous Data Fusion

29 29 03 Multi-source Heterogeneous Data Fusion The large amount of data is a big challenge Volume Refers to the speed requirement for processing data Velocity Energy Big-data Variety Variety means the increasing complex of data types Value Data itself is meaningless unless valuable knowledge

30 30 03 Multi-source Heterogeneous Data Fusion System heterogeneous data are stored in different system or database Semantic heterogeneous two records of the same entity have different express Heterogeneous data Structured heterogeneous the data are not only structured data but also semistructured data and unstructured data Grammar heterogeneous data have different formats such as units of data

31 31 03 Multi-source Heterogeneous Data Fusion 1. Semantic heterogeneous data fusion (1) Duplicate database records: name date of birth terminal name terminal address power K.X. Huang Taoyuan K.X. Huang Taoyuan (2) Fusion methods 1 Field Matching Method 2 Sorted-neighborhood Method

32 32 03 Multi-source Heterogeneous Data Fusion 1 Field Matching Method(FMM) Calculate the weight of each attribute of records Weight Similarity Calculate the similarity between records Records matching and detection Detection

33 33 03 Multi-source Heterogeneous Data Fusion a) Weight Y = y 1, y 2,, y n, where y i = y i1, y i2,, y im, i 1,2,, n Y means n records in a database with m attributes T ik T i1, T i2,, T im, i 1,2,, N, T ik 1 T = T 1, T 2,, T m, where T k = N T Τ ik N, k 1,2,, m σ i=1 T k T k, S = max T 1, T 2,, T m, Convert T k to an integer T k S 1 W k = 1 S i i=tk, k 1,2,, m W j = m W j k=1 W k, j 1,2,, m

34 34 03 Multi-source Heterogeneous Data Fusion b) Similarity y ik = y i1, y i2,, y ip y jk = y j1, y j2,, y jq the k-th attribute of the record y i has p strings the k-th attribute of the record y j has q strings q sim y ik, y jk = max score y ika, y jkb p, a 1,2,, p b=1 m sim y i, y j = W k sim(y ik, y jk ) c) Detection k=1 correct numbers completeness = detected numbers correct numbers precision = true numbers

35 35 03 Multi-source Heterogeneous Data Fusion 2 Sorted-neighborhood Method(SNM) Sort current window Window Detection next window

36 36 03 Multi-source Heterogeneous Data Fusion (3) Example Power Data of Taoyuan residential community name date of birth terminal name terminal address power K.X. Huang Taoyuan K.X. Huang Taoyuan D.J. Zhang Taoyuan D.J. Zhang Taoyuan Results of two methods Detected number correct number completeness precision time(s) FMM SNM

37 37 03 Multi-source Heterogeneous Data Fusion 2. System heterogeneous data fusion (1) Data are stored in different database (2) Fusion method Open Database Connectivity (3) Example Excel Text SQL ODBC Oracle database

38 Conclusions and Future Work 38

39 04 Conclusions and Future Work 39 Conclusions 1. Integrated energy system has drawn widely attention around the world. Researches from various of fields greatly promote the development of Integrated energy system. 2. Energy Big-data Platform is the foundation of an Integrated Energy Platform, and is a significant research field. 3. Methods for multi-source heterogeneous data fusion is introduced, furthermore, establishment of an Energy Bigdata Platform framework is on going.

40 04 Conclusions and Future Work 40 Future Work An Integrated Energy Platform based on the Energy Big-data Platform Integrated Energy Platform Energy Big-data Platform Integrated Energy Planning Platform Integrated Energy Operation Platform Integrated Energy Trading Platform

41 Thanks for Your Attention! 41

20775A: Performing Data Engineering on Microsoft HD Insight

20775A: Performing Data Engineering on Microsoft HD Insight Duration: 5 days; Instructor-led Implement Spark Streaming Using the DStream API. Develop Big Data Real-Time Processing Solutions with Apache