Ein Rennen der anderen Art: Big-Data Plattformen im Automobilbau

Size: px

Start display at page:

Download "Ein Rennen der anderen Art: Big-Data Plattformen im Automobilbau"

Dylan Quinn
6 years ago
Views:

1 Ein Rennen der anderen Art: Big-Data Plattformen im Automobilbau Thomas Pagel, Principal Technologist, Data & Analytics Franziska Weng, Consultant, Data & Analytics 2017 Avanade Inc. All Rights Reserved.

2 Agenda The Avanade Story How we do Analytics in the Cloud Proof of Concept Overview Analytical Processing Results/Lessons Learned Q&A

3 The Avanade Story Innovative digital services, business solutions and design-led experiences delivered through the power of people and the Microsoft ecosystem

4 Avanade by the numbers (FY16) Created in 2000 as a joint venture between Accenture and Microsoft today majority owned by Accenture. We help clients to maximize their performance and realize their vision. Our innovative solutions help to improve productivity and efficiency. $ 2,4 B Sales 20 % Average annual growth Client partners worldwide typically mid to large-scale enterprises and government agencies 43 % Of Global 500 companies as clients 4

Avanade group in Germany, Austria and Switzerland 19

industry expertise to build and deploy solutions to

5 Avanade group in Germany, Austria and Switzerland 19 locations with more than 850 employees The Avanade group includes: Avanade Infoman KCS.net Our professionals combine technology, business and industry expertise to build and deploy solutions to realize results for clients and their customers. Bratislava 1 Nearshore development center in Slovakia

6 Avanade Global Delivery Network: fast efficient development and delivery of quality services More than professionals in over 80 locations

7 Recognized for great work as well as being a great place to work Avanade s innovative spirit, leadership and commitment to high ethical standards have resulted in a broad range of industry honors. Industry Recognition Top Employer Recognition Microsoft Auszeichnungen: Alliance SI Partner of the Year 2016, 2015, 2014, 2013, sowie 8 weitere seit 2001 Microsoft Customer Relationship Management (CRM) Partner of the Year 2015 Dynamics Inner Circle and President s Club 2015, 2014, 2013, 2012, sowie 6 weitere seit 2005 Country Partner of the Year 2015 Italien, 2013 Deutschland und Spanien Microsoft Dynamics Reseller of the Year 2015 Mobility Partner of the Year 2014, 2013, 2012 Microsoft Service Partner of the Year 2013

8 How we do Analytics in the Cloud

Avanade s Modern Analytics Platform (AMAP) - an industrialized Managed Service for IoT/Analytic solutions Any Data from any Device or any Resource in any Format can be loaded in a secure way in Azure

9 Avanade s Modern Analytics Platform (AMAP) - an industrialized Managed Service for IoT/Analytic solutions Any Data from any Device or any Resource in any Format can be loaded in a secure way in Azure IoT/ Analytic Platform AMAP Built on Microsoft Azure PaaS components Azure Data Platform + Azure ML + Power BI Combines the shared service labor cost model with consumption based compute cost Provides connectivity for Industurial Devices and Business Integration Use Client s Microsoft EA for Azure 9

10 Avanade Modern Analytics Platform Capabilities 10

11 Avanade s sophisticated Analytic platform gets you there Challenges Outcomes Traditional Approach The AMAP Approach Combine real-time and advanced analytics with existing systems & data Accelerate Insights Pressure is on to realize results Innovate & Accelerate Insights Advanced Analytics Bridge the skills gap on advanced techniques Provide the business the data and analytics platform capabilities Manage, upkeep and running of existing systems Augment your Talent Pool Ready on Demand Answer Business Questions...Meanwhile customer needs are changing Self-Service & Discovery Data Management Big Data/IOT/ Streams Develop & Design Experiences Deliver Digital Platforms Support & Enablement Hire / Train People Buy Hardware & Software Set Strategy, Goals & Questions Innovate & Accelerate Insights AMAP Managed Services Set Strategy, Goals & Questions Avanade Modern Analytic Platform (AMAP) is a Platform and a Managed Service Powering new solutions and offerings including Digital Marketing Analytics, IOT Analytics, Self-Service Analytics and custom client solutions Staying out front Quick and Comprehensive 11

12 Proof of Concept Overview

13 The PoC: Azure vs. AWS vs. Teradata The customer asked three service providers to demonstrate the capabilities of their teams and their preferred platform: Avanade/Accenture on Microsoft Azure Another Consulting Company on Amazon Web Services Teradata with their ASTER Appliance

Introduction: Camshaft drive chain failure

14 Introduction: Camshaft drive chain failure prediction 1/2 Situation: A German car manufacturer has frequent issues with a specific engine in regards to failure and extension of the camshaft drive chain leading to engine failures and frequent workshop visits.

Introduction: Camshaft drive chain failure prediction 2/2 Goals: A German car manufacturer wants to identify potential camshaft issues early

deposition, climate condition, too long service intervals) Business goals: Predict warranty cases save costs/increase customer satisfaction

15 Introduction: Camshaft drive chain failure prediction 2/2 Goals: A German car manufacturer wants to identify potential camshaft issues early to do predictive maintenance Analytics goals: Predict extension of camshaft drive chain lengthening and potential causes (oil, soot deposition, climate condition, too long service intervals) Business goals: Predict warranty cases save costs/increase customer satisfaction through early replacement of the camshaft drive chain without impact on the drivers experience IT goals: Evaluate Azure Big Data Analytics stack as well as TCO

identify reasons for camshaft drive chain failures?

16 Analytical Questions Is it possible to select features relevant for analytics processes? predict camshaft drive chain failures? identify reasons for camshaft drive chain failures? make a statement about sensorreadings regarding starting chain lengthening?

17 Analytical Processing

Big Data Analytics Case Approach using the Azure platform

briefing workshop Set - up Data sourcing & data management

Analytical Solution Discuss analysis pilot goals & scope

Define analysis pilot goals & scope Setup infrastructure

Data quality checks Build data structure OUTCOMES

test data Presentation of proof of concept results and

case briefing done Analytical data transferred Analytical

18 Big Data Analytics Case Approach using the Azure platform capabilities to realize analytics results fast Analytical briefing workshop Set - up Data sourcing & data management Insight generation and validation Presentation of Analytical Solution Discuss analysis pilot goals & scope Identify relevant data sources Basic data understanding Define analysis pilot goals & scope Setup infrastructure Data transfer Data understanding Design data structure Data quality checks Build data structure OUTCOMES Calculate models and build reports Evaluate models with test data Presentation of proof of concept results and potential next steps Recommendations for the client Use case briefing done Analytical data transferred Analytical data structure ready Analytical results ready Final use case presentation and next steps roadmap

19 Proof of Concept: Sequence File to Text Conversion Apache Spark is an open-source cluster-computing framework based on Hadoop technology. You can use it interactively from the Scala, Python and R shells. import org.apache.hadoop.io._ val fn = "<filename>" val sf2 = sc.sequencefile[mapwritable,byteswritable](fn) val xml = sf2.map(row => org.apache.hadoop.io.text.decode(row._2.getbytes())) Additionally we needed to use Hive to clear line breaks from the text formatted XML data. INSERT INTO TABLE <target table> SELECT REGEXP_EXTRACT( REGEXP_REPLACE( REGEXP_REPLACE( <column name of the sequence file results table>, '\\n', ''), '\\r', ''), '^.*(<\\?xml.*)', 1) FROM <sequence file result table>;

20 Proof of Concept: XML Text to Table Structure Conversion A good way to convert terabytes of complex XML to table structured data is applying XSLT using an HDInsight Apache Spark Cluster and Scala. import com.elsevier.spark_xml_utils.xslt.xsltprocessor val xmlkeypair = sc.sequencefile[string, String]( <filename of xml data>") val stylesheet = sc.textfile( <filename of xsl>").collect.head val srctitles = xmlkeypair.mappartitions(recsiter => { }) val proc = XSLTProcessor.getInstance(stylesheet) recsiter.map(rec => proc.transform(rec._2))

Proof of Concept: Clean and Preprocess Data Azure SQL Data Warehouse is the technology you should choose, because: + You can stop it over night and pay less.

21 Proof of Concept: Clean and Preprocess Data Azure SQL Data Warehouse is the technology you should choose, because: + You can stop it over night and pay less. + It is about as fast as an HDInsight Apache Spark Cluster. + You can use PolyBase to comfortably load data from Azure Blob Storage. Cleaning and Preprocessing Tasks performed in Azure DWH: 1. Load data using PolyBase 2. Unique car filtering 3. Relevant sample selection 4. Target variable generation Load from external Table (PolyBase) Relevant sample selection Generate target variable 21

22 Visual Data Inspection using Power BI Visual data inspection can be done using Power BI. Additionally, the Quick Insights feature can show you some interesting information. (

23 23 Analytical Processing using Azure Machine Learning Root-Cause Analysis Feature data and Label from same point in time Which features behavior is related to the camshaft chain failure? Predictive Analysis Feature data from one week before Label data Which features past behavior is related to the camshaft chain failure? Format adjustment Modeling Target variable labeling Column selection Sample splitting Pearson Correlation Analysis

24 Sample Results (most important Features/Starting Values)

25 Sample Results (Prediction/Root Cause Analysis)

26 Results/Lessons Learned

27 Guess who won?

28 It took the client 2 years to archive the same quality of results Avanade/Accenture provided after 2 weeks analytical work

29 Business Goals - Conclusion Success Dimension The Azure platforms unique proposition Cost Flexibility Security Productivity You pay only for resources that you are actually using - cost are transparent on a monthly bill. The architecture is highly flexible The client is able to decide on the toolset and also can bring in own technologies as Virtual Machines (IaaS) if necessary. Azure is using market leading and regulator compliant security to keep client s data safe in motion, in rest and on use. Additional security layers and management & monitoring can be provided by Accenture. Instead of spending time and money on installation, operations and management, the platform is handled as a service. The client s resources can spend their time on their core competency to drive value.

Key Lessons Learned Around the project: Data provisioning can take quite a while Complex XML -> Complex

never happen Around Azure: Azure enables agile projects fail fast, learn fast Find the appropriate tools to tackle

and competitive Azure ML is powerful and easy but lacks some control (better with R) Azure SQL DWH was already

30 Key Lessons Learned Around the project: Data provisioning can take quite a while Complex XML -> Complex transformation/analytics Include the business know-how Test your assumptions/hypothesis No blind believe in will never happen Around Azure: Azure enables agile projects fail fast, learn fast Find the appropriate tools to tackle the challenge Azure often offers more than one tool for the same purpose Even on Hadoop Azure is very up-to-date and competitive Azure ML is powerful and easy but lacks some control (better with R) Azure SQL DWH was already great as a preview PaaS offerings are often a great alternative vs. IaaS based solutions Big Data is not necessarily equals Hadoop

31 Questions? Just visit us at our booth!

32 Thank you

Azure Data Analytics & Machine Learning Seminar. Daire Cunningham: BI Practice Area Manager

Azure Data Analytics & Machine Learning Seminar Daire Cunningham: BI Practice Area Manager AGENDA 09:00 AM 09:30 AM Registration & Refreshments 09.30AM 10:00 AM 10:00 AM 10:30 AM Welcome & Keynote, Ger