Internet of Things. Dilip Kumar Kari

Transcription

1 Internet of Things Dilip Kumar Kari

2 Agenda Problem Statement IOT in Oil fields Describing the use case Real-time operations monitoring Seismic survey in deep oceans The Edge Protocols Shipping the data Data processing frameworks Scalable Messaging Systems Processing the data Real-time Distributed Processing Frameworks Storage Visualization Demo Questions

3 Create New Requirements Businesses need trusted, real-time insights from full-fidelity data Applications need access to both data in motion and data at rest IT needs to capture multi-directional IoT data flows from point to point Challenge: the IoT perimeter is jagged and outside the data center

4 Collecting and Analyzing IOT Data IOT + Predictive Analytics affects all industries BI vs Data Science approach BI (traditional and doing for decades) only structured data Descriptive analysis Standard Reporting What happened? Data Science Predictive analysis prescriptive analysis (guidance of what to do on predicted outcome) Why? what will? what should id do?

5 Data Science Process Define Hypothesis to test or prediction to be made Gather data DL (SQL + Hadoop) Build Schema (schema on read/query) Visualize the data Build Analytical Model (SAS, MAtLab, MADLib, Mahout, R ) Evaluate the model results (probabilities, confidence levels) - what is the output to different input params as they change

6 Today s Architectural Limitations Data movement software was built for standardized data and one-way flows Tools for unstructured data are customized, disjointed and difficult to manage Such tools cannot securely collect, conduct and curate IoT data flows, which are multi-directional from point to point at the perimeter Today s architectures limit potential insight by making it difficult and expensive to manage IoT data both in motion and at rest

7 The Use cases A large Oil & Gas services company is monitoring deep sea and onshore drilling jobs across the world from Houston. A seismic survey company is generating raw data for oceanic surveys on behalf of BP, Shells, Exxon s of the world.

8 Oilfield Data Collection OILFIELD OPERATIONS 1000s of sensors per site Temperature, Pressure, Flow, Pumps, Actuators, Valves, etc. REMOTE COMMS DEVICES RTUs, PLCs, Instrumentation Systems DATA LINK Satellite, Cellular, Radio SCADA NETWORK OPC Servers for a large basin DATA LINK Satellite, Cellular, Radio HISTORIA N/s Time series logs and history DATA LINK Corporate WAN CORP SYSTEMS & HADOOP Systems of Record & Analytics Platform ANALYTICS Real Time Data + All Other Datasets

9 Data from the field, factory floors and plants

10 Technologies that enables IoT Bluetooth zwave is the protocol used in most smart home applications LTE - communication specification for mobile networks Wifi-Direct - peer to peer communication without need to have an access point Open Standard Protocols like OPC UA/DA

11 Shipping the Data AG. Image source " Siemens AG 2015, All rights reserved"

12 Apache Nifi PRIORITIZ E GATHE R DELIVE R RECALIBRATE Track from the edge (well site, equipment, etc.) Through the datacenter (and back) It s not just how quickly you move data it s about how quickly you can change behavior and seize new - Guarantee high fidelity data delivery and eliminate data loss even with intermittent communications - make decisions from credible and reliable data - Provide secure data collection and movement from data generation to data rest with 21 encryption algorithms - Enable comprehensive provenance and lineage with a complete digital chain of custody for real time multidirectional data streams, flow specific quality of service, and visibility into valuation of your field data capture infrastructure - Allow for real time prioritization of data and interactive changes to the data flow with visual command and control

13 RPC & OPC Specifications COM, DCOM, OLE Classic OPC (Open Platform Communication) OPC XML DA (Data Access) OPC HDA (Historical Data Access) OPC AE (Alarm & Events) OPC Xi (Express Interface) OPC UA

14 OPC UA OPC UA differs significantly from its predecessor ü International Standard IEC ü Multi-platform SOA based implementation, ü including portable ANSI C, Java and.net implementations ü Scalability: from smart sensors and smart actuators to mainframes (enterprise systems) ü Multi-threaded, as well as single-threaded/single-task operation necessary for porting the stack to embedded devices ü Security, based on new standards ü Configurable time-outs for each service ü Chunking of big datagrams ü Model complex data structures for collaboration with other standard organizations

15 Distributed Intelligence based on standards OPC UA standard for enterprise to connect DCS, ERP, MES, SCADA, PLC and Sensors

16 SCADA SCADA (Supervisory Control And Data Acquisition) is a system for remote monitoring and control that operates with coded signals over communication channels (using typically one communication channel per remote station). It is a type of industrial control system (ICS) SCADA systems historically distinguish themselves from other ICS systems by being large-scale processes that can include multiple sites, and large distances These processes include industrial, infrastructure, and facility-based processes

17 Choosing an Architecture Do Lambda OR Kappa?

18 Lambda Architecture

19 Kappa Architecture

20 IOT Architecture Broker Push Push or Pull OPC Server Producer Consumer + Spout Storm Bolt(s) Bolt(s) Region Server HDFS + Hive

21 SCADA IoT Demo

22 Questions