Using MRG and Infinispan for Large Scale Integration. Prajod Vettiyattil

Transcription

1

2 Using MRG and Infinispan for Large Scale Integration Prajod Vettiyattil 2

3 What this session is about Challenges in Large Scale integration Use cases for Large Scale integration How to solve the challenges Solution Products to implement the solution The Open Source difference Challenges Use Cases Solutions 3

4 4 Key Phrases

5 Phrases: 1 Large Scale Integration Integration of 10s or 100s of systems, and exchange GBs of messages in a day Big Data A changing threshold Data in the Terabytes, Petabytes, Exabytes Asynchronous Messaging Message oriented middleware Real time systems Systems that are built to respond to requests in real time, with predicable, consistent response times 5

6 Phrases: 2 Grid A set of interconnected computers that work in parallel to solve a computing problem Cloud Computing Computing as a service Client of the cloud is isolated from the details of the implementation of the service 6

7 Challenges Challenges Use Cases Solutions 7

8 The Key Challenges Large number of systems 8

9 The Key Challenges Complexity of connection between these systems 9

10 The Key Challenges Constraints on the systems and on the connections soap / http csv / ftp Rest / http csv / file Adapte r Text/tc p soap / http csv / ftp Adapte r soap / http csv / ftp Rest / http csv / file Adap ter csv / file Adap ter Adap ter soap / http csv / ftp Rest / http csv / file Adap ter csv / file Adap ter Rest / http csv / file Adapte r soap / http csv / ftp Rest / http csv / file Adapte r 10

11 Use Cases and Solutions Challenges Use Cases Solutions 11

12 Architecture Wireless Telco BSS Integration Portals, Front End System, Partner Gateways Process Automation/ Business Process Administration and Monitoring Services BSS Applications Infrastructure S ervices for Middleware Mediation Provisioning CRM Workforce Interconnect Transport Services Revenue Caching Services Billing Middleware Message Broker Enterprise Service Bus Load Balancing Services Fault Failover Services Number Inventory Element JEE Server Recovery Services Security Services Telco Network S ystems 12

13 Telco scalability Some requirements 75+ million customers Plan for Terabytes of CDRs and other messages per day Performance is critical to customer experience and retention CRM, Billing, Mediation, Middleware, Data warehouse 13

14 High volume use case 1 CRM to Billing Integration Fraud CRM Network Switch Mediation Billing Middleware Other BSS Apps Technical Requirements Memory Threads Sockets Sender performance Receiver performance Data Warehouse Point to Point Connection 14

15 High volume use case 1 CRM to Billing Integration: with Middleware Infrastructure Network Switch Mediati on Fraud Billing Data Warehouse CRM Middleware Other BSS Apps Fraud CRM Network Switch Mediation Billing Middleware Other BSS Apps Data Warehouse Middleware Infrastructure Caching Load Balancing Failover Recovery 15

16 Middleware Infrastructure expanded Fraud Managem en t CRM Netw ork Switc h Med iatio n Billin g Middlewa re Oth er BSS Ap p s Data Warehou se Middl eware In frastr ucture Middleware High Speed, Reliable Massaging Middleware Infrastructure Compute Node Scaling Work Load Resource Distributed Caching Failover Recovery Cluster Toolkit 16

17 Middleware Infrastructure Products Netw ork Switc h Med iatio n Fraud Managem en t Billin g CRM Middlewa re Oth er BSS Ap p s Middleware Any ESB Any JEE Server Any Message Broker Data Warehou se Middl eware In frastr ucture MRG Grid High Speed, Reliable Massaging Middleware Infrastructure Compute Node Scaling Work Load Resource Distributed Caching Failover Recovery Cluster Toolkit 7 6 MRG Messaging Infinispan 17 8

18 Middleware Infrastructure: Products MRG Messaging Fraud Managem en t CRM Netw ork Switc h Med iatio n Billin g Middlewa re Oth er BSS Ap p s MRG Messaging Data Warehou se High Speed, Reliable Massaging Distributed Caching Middl eware In frastr ucture Middleware Infrastructure Compute Node Scaling Failover Work Load Recovery Resource Cluster Toolkit AMQP support Native RDMA, Infiniband Can use MRG Realtime Large message support(> GB) Clustering and Failover High speed, journal based persistence Java and C++ brokers Based on Apache Qpid 18

19 Middleware Infrastructure: Products MRG Grid Fraud Managem en t CRM Netw ork Switc h Med iatio n Billin g Middlewa re Oth er BSS Ap p s MRG Grid High Speed, Reliable Massaging Distributed Caching Data Warehou se Middl eware In frastr ucture Middleware Infrastructure Compute Node Scaling Failover Work Load Recovery Resource Cluster Toolkit Scalable Grid Scheduler Resource variety: Desktop to Cloud schedulers Low latency results: Using MRG Messaging Dynamic provisioning High Availability Grid Federation Is based on the Condor Grid project 19

20 Middleware Infrastructure: Products Infinispan Fraud Managem en t CRM Netw ork Switc h Med iatio n Billin g Middlewa re Oth er BSS Ap p s Infinispan Data Warehou se High Speed, Reliable Massaging Distributed Caching Middl eware In frastr ucture Middleware Infrastructure Compute Node Scaling Failover Work Load Recovery Resource Cluster Toolkit In memory Data Grid Distributed cache Peer to peer communication between nodes Flexible persistence: JDBC, File, Amazon S3 Map reduce: node local computing Implementation for performance 20

21 High volume use case 2 Post Trade Securities Processing Sources Trading Applications Data Processing Solution Processing Nodes Targets Accounting Solution Node1 Node2 Post Trade Files File Splitter + Load Allocator Aggregator Output Channels Node3 Node4 Trading Applications Data Services Risk Solution 21 Ref Data Solution Src 3 Src 4 Customer Master

22 Post Trade Securities Processing Technical Requirements Processing Nodes Node1 Node2 Input Channels File Splitter + Load Allocator Node3 Node4 Aggregator Output Channels Technical Requirements File Streaming Multiple Data views, Data sources Data Aggregation Reliable Messaging Data Services 22

23 Post Trade Securities Processing Products 4 Infinispan MRG Messaging 5 Processing Nodes Node1 Node2 6 Input Channels File Splitter + Load Allocator Node3 Node4 Aggregator 3 Output Channels 7 Data Services JBoss Data Services 1 MRG Grid 2 23

24 MRG Realtime Consistent, predictable response Websphere Realtime: RTSJ Messages/microsecond Response Time Source: Red Hat 24

25 A Recap of the Solution Challenges Use Cases Solutions 25

26 Challenges, Solutions and Products Challenge Solution Products 1 Small data elements, high volume Distribution, load balancing and partitioning 2 Large data elements File splitting, distribution, in place processing 3 Data views, Many data sources Data Services 3 Predictability Real time kernels, real time JVMs 4 Availability Load balancing, clustering, failover 5 Reliability File based caches, DB persistence MRG Messaging, MRG Grid MRG Messaging, MRG Grid JBoss Data Services MRG Realtime, RTSJ MRG Grid, Infinispan MRG Messaging, Grid, Infinispan 6 Scalability Compute grids, Data grids, Asynchronous messaging MRG Messaging, Grid, Infinispan 26

27 27 Solution Alternatives

28 The Map Reduce method Split data, process in parallel, aggregate results Input Data Map Phase Task Tracker Name Node Reduce Phase Task Tracker Split Data Data Output Data Client Task Tracker Task Tracker 28 Job Tracker

29 GT3 and Condor Globus Toolkit Open source toolkit for Compute Grids Architecture, Service Model and Implementation Job Tracking,, Monitoring, Resource Data : Movement, Location Registry Condor 29 Grid Framework from University of Wisconsin Compute Node Scaling Job Scheduling Idle time utilization

30 Commercial Tools IBM IBM Cloudburst Websphere Virtual Enterprise Websphere Realtime Oracle Oracle Exalogic Oracle Coherence Oracle Grid Engine(Sun Grid Engine) Terracotta Quartz Big Memory 30

31 31 The Open Source difference

32 The Advantages Smaller adoption steps to reduce risk Flexible Cost Model Subscription based pricing Incident based pricing Cloud alignment Elastic pricing model Cloud Software platforms use open source Innovation from wider community Custom enhancements 32

33 33 Conclusion

34 Key Points Discussed Large Scale Integration Impact of Big Data on Integration Use cases Telecom Securities Solutions MRG Infinispan Data Services Open source differentiators 34

35 35 Questions