CI in the cloud. Case Talend & Nordcloud Ilja Summala, Group CTO, Nordcloud Anubhav Sharma, Senior Software Architect, Talend Deutschland

Size: px

Start display at page:

Download "CI in the cloud. Case Talend & Nordcloud Ilja Summala, Group CTO, Nordcloud Anubhav Sharma, Senior Software Architect, Talend Deutschland"

Brian Parrish
5 years ago
Views:

1 BERLIN 2015

2 CI in the cloud Case Talend & Nordcloud Ilja Summala, Group CTO, Nordcloud Anubhav Sharma, Senior Software Architect, Talend Deutschland

3 About Nordcloud Nordcloud is leading European AWS consultancy with offices in Helsinki, Munich, London, Stockholm, Oslo and Malmö. Nordcloud is AWS Premier Consultancy Partner for We have completed over 200 cloud migration projects.

4 Talend Challenge Talend provides data integration software Introducing cloud SaaS to complement onpremise software Nordcloud helped Talend to build secure, scalable and and automated service platform on AWS

5 AWS toolset for SaaS providers Elastic Capacity EC2 ELB Route 53 Cloudformation Security VPC Security Groups Subnet NACLs KMS Cloudtrail Automation Cloudformation /Troposphere / Boto [Configuration management tools Puppet/Chef] [CI tools (Jenkins)]

7 Connecting the Data-Driven Enterprise Key Facts because Data-Driven companies have 23 times greater customer acquisition 6 times greater customer retention 19 times more profitability

8 A Data-Driven world Big Data Cloud Connected Enterprise Self-Service Hadoop becomes the dominant data processing platform Cloud overtakes on-premises investment APIs Connect the Data-Driven World Self-Service becomes the norm

9 Unified Integration Platform Lowest Cost of Ownership Open, Standards Based Run in the Cloud, On-premises or Hybrid Big Data Leadership

10 Broad Customer Base

11 Talend Integration Cloud

$Enable big data insight at a fraction of the cost Empower$

12 Talend Integration Cloud Simplify and accelerate cloud integration Manage hybrid integration with instant, elastic and secure capacity. Enable big data insight at a fraction of the cost Empower anyone to integrate data using a self-service interface

13 Example Use Case Customer Data Warehouse Background Lacking insight into sales and marketing programs Goals 360 customer view Weekly pipeline snapshots Full lead-to-deals won lifecycle Redshift

14 Use Cases Connecting SaaS-SaaS, SaaS-Ground Internet of Things Self-Service Integration Offload to the cloud Analytics in the cloud Redshift

Data Integration Data Quality Big Data Integration 800+ Components

15 Talend Integration Cloud Architecture A secure cloud integration platform Talend Studio DESIGN MANAGE MONITOR Talend Cloud Engine Web Interface Data Integration Data Quality Big Data Integration 800+ Components Real-time Integration (Winter) API Gateway (Winter) Talend Remote Engine On-Premises

flows Enforce governance and security Publish to the cloud

16 Everything you can do in Studio, now works in the cloud! Capabilities Productive graphical tooling Create simple or complex flows Enforce governance and security Publish to the cloud Benefits Faster time-to-market Lower costs Establish control Integration Specialist

Empower Others with Integration Self- Service

agility Reduce error rates Empower a new class

17 Empower Others with Integration Self- Service Capabilities Design anywhere Extract, cleanse and load data Reuse components published from Studio or Talend Exchange Benefits Increase agility Reduce error rates Empower a new class of users Ensure data governance and security Business User

integration actions SaaS Big data Packaged apps Technology

18 Speed Project Delivery with Talend Exchange Leverage the power of Talend s open source community Hundreds of pre-packaged integration actions SaaS Big data Packaged apps Technology connectors Data quality Easily add hundreds of additional components from Talend Studio

19 Talend Integration Cloud on AWS Manage multiple environments

20 Challenges Picking up java artifacts and turning them into working stacks. Handling multiple teams and Environments Dev, QA, staging, production. Single vs. Multiple AWS accounts. Orchestration. Dealing with AWS KMS region dependency. QA-Auto-RT Dev latest QA-Auto-UI Staging Production Dev Ops

21 Package everything into rpms. Use S3 as repository to store rpm s. Single entry point for packages.

Single AMI for multiple stacks & environments. One ami for every role. Instance started as part of the cloud formation stack. Subenv parameter is used to build multiple stacks.

22 Single AMI for multiple stacks & environments. One ami for every role. Instance started as part of the cloud formation stack. Subenv parameter is used to build multiple stacks. User data is generated by troposphere and facts are included. Puppet converts the userdata into custom facter facts. These facts are used for hiera lookup for that particular subenv.

24 Hiera look-up What classes to load? Which is my java version? Where is my mongodb?

25 Node signature Node signature is the string representation of the instance t_facts The node signature variable within our configuration management system contains list of all instances which have the same signature

Problem: EC2 instances needs to find each other within the same stack. (service discovery) Orchestrate instance startup based on fine grained service availability.

26 Problem: EC2 instances needs to find each other within the same stack. (service discovery) Orchestrate instance startup based on fine grained service availability. Solution: Stack membership - serfdom. Gossip based cluster membership. Every instance has a tag and tfacts. Serf facts are assigned using facter facts. Serf agent on every instance is queried for discovery. Distributed dependency and sequence of service startups handled using Amazon wait condition on cloud formation level.

27 Hiera look-up Node signature variables are interpolated in Hiera to list of nodes that have the same signature.

28 Keys in KMS are region specific and only available in that region. To handle disaster recovery and move to different regions the application architecture should be designed to incorporate this factor. We use region specific master key but region independent data key to encrypt application data.

29 Project Partnership &

Talend: Nordcloud Partnership Scope of Engagement AWS Consulting / System Setup General

config) Backup / Recovery / High Availability Day to Day Operations AWS Helpdesk

for critical incidence on a 24/7 base AWS Partner Consolidated Billing AWS Enterprise

30 Talend: Nordcloud Partnership Scope of Engagement AWS Consulting / System Setup General setup of the entire stack in a fully automated way Special expertice (MongoDB Cluster config) Backup / Recovery / High Availability Day to Day Operations AWS Helpdesk On-side support on day to day operations Centralized System Monitoring Stand-by support for critical incidence on a 24/7 base AWS Partner Consolidated Billing AWS Enterprise Support AWS cost management / optimization (e.g Instance Type Selection, Reserved Instance Mgmt) &

31 Talend: Nordcloud Partnership Engagement Milestones After a final selection period In November 2014 We agreed to start with Nordcloud as our partner for Cloud Operations support. Start: December 2014 (Primary Reasons: Used to work with same tech stack (Cloud Formation / troposhere, puppet, mongodb and others ) Senior consultants for on side support with a track record in Cloud Operations System Setup Consulting Working closely with our DevOps Team to setup the system to a full production scale deployment (incl. 3 Beta Phases / Deployments) Production Start (End Apr 15) Continues Setup / Config Support for next version Support our CloudOps team on day to day operations Next: Rolling Release Update from Spring 15 (1.0) version to Summer 15 Version (1.1) end of June Continues day to day operations support Dec 15 Dec - Apr 15 Apr 15 Jun &

Financial Services Industry in AWS

Financial Services Industry in AWS Ilja Summala Group CTO, Nordcloud About Nordcloud and Me Nordcloud in Nutshell 100% Public Cloud 300+ employees - one of the fastest growing tech companies in Europe