KRISTINA BLAŠKOVÁ SOA & ETL

Transcription

1 KRISTINA BLAŠKOVÁ SOA & ETL

2 SOA & ETL SOURCES Josuttis, Nicolai. (2007). SOA in Practice: The Art of Distributed System Design (Theory in Practice). Kimball, Ralph, Caserta, Joe. (2004). The Data Warehouse ETL Toolkit: Practical Techniques for Extracting, Cleaning, Conforming, and Delivering Data. Both available on libgen.io

3 SOA & ETL CONTENT SOA What is architecture? How is SOA different? SOA manifesto Key notions Technical concepts of SOA ETL Motivation Extract Clean Transform Load

4 SOA - SERVICE ORIENTED ARCHITECTURE 4 SERVICE ORIENTED ARCHITECTURE Software architecture is a coherent set of abstract patterns guiding the design of each aspect of a larger software system. Software architecture underlies the practice of building computer software. In the same way as a building architect sets the principles and goals of a building project as the basis for the draftsman's plans, so too, a software architect or systems architect sets out the software architecture as a basis for actual system design specifications, per the requirements of the client.

5 SOA - SERVICE ORIENTED ARCHITECTURE 5 SERVICE ORIENTED ARCHITECTURE

6 SOA - SERVICE ORIENTED ARCHITECTURE 6 SERVICE ORIENTED ARCHITECTURE IT realization of some self-contained business functionality

7 7 WHAT? SERVICE ORIENTED ARCHITECTURE PHILOSOPHY

8 SOA - SERVICE ORIENTED ARCHITECTURE 8 SOA DEFINITIONS SUMMARY SOA is an architectural paradigm for dealing with business processes distributed over a large landscape of existing and new heterogeneous systems that are under the control of different owners.

9 SOA - SERVICE ORIENTED ARCHITECTURE 9 SOA MANIFESTO - Business value over technical strategy Strategic goals over project-specific benefits Intrinsic interoperability over custom integration Shared services over specific-purpose implementations Flexibility over optimization Evolutionary refinement over pursuit of initial perfection That is, while we value the items on the right, we value the items on the left more.

10 SOA - SERVICE ORIENTED ARCHITECTURE 10 WHAT WE WILL GO THROUGH Characteristics of large distributed systems Technical concepts of SOA: Interoperability Services Loose coupling

11 SOA - SERVICE ORIENTED ARCHITECTURE 11 CHARACTERISTICS OF LARGE DISTRIBUTED SYSTEMS Legacy Heterogeneity Complexity Different owners Imperfection Redundancy Bottlenecks

12 TEXT 12 THE KEY TECHNICAL CONCEPTS OF SOA services interoperability loose coupling

13 KEY TECHNICAL ASPECTS OF SOA: INTEROPERABILITY, SERVICES, LOOSE COUPLING 13 MAGIC BUS

14 KEY TECHNICAL ASPECTS OF SOA: SERVICES, INTEROPERABILITY, LOOSE COUPLING 14 THE TERM SERVICES Essentially, SOA is a software architecture that starts with an interface definition and builds the entire application topology as a topology of interfaces, interface implementations and interface calls. SOA would be better-named interfaceoriented architecture. -> Everything that can be used as an interface, and represents a self-contained business functionality can be a service.

15 KEY TECHNICAL ASPECTS OF SOA: SERVICES, INTEROPERABILITY, LOOSE COUPLING 15 Technically driven interface Business driven interface

16 KEY TECHNICAL ASPECTS OF SOA: SERVICES, INTEROPERABILITY, LOOSE COUPLING 16 ADDITIONAL SERVICE ATTRIBUTES Stateless Coarse grained or composable Self- contained Discoverable Idempotent

17 KEY TECHNICAL ASPECTS OF SOA: SERVICES, INTEROPERABILITY, LOOSE COUPLING 17 LOOSE COUPLING Asynchronous communication

18 KEY TECHNICAL ASPECTS OF SOA: SERVICES, INTEROPERABILITY, LOOSE COUPLING 18 LOOSE COUPLING Heterogenous datatypes

19 KEY TECHNICAL ASPECTS OF SOA: SERVICES, INTEROPERABILITY, LOOSE COUPLING 19 LOOSE COUPLING Mediators

20 KEY TECHNICAL ASPECTS OF SOA: SERVICES, INTEROPERABILITY, LOOSE COUPLING 20 LOOSE COUPLING Mediators Compensation

21 KEY TECHNICAL ASPECTS OF SOA: SERVICES, INTEROPERABILITY, LOOSE COUPLING 21 LOOSE COUPLING Mediators Compensation Control of process logic

22 KEY TECHNICAL ASPECTS OF SOA: SERVICES, INTEROPERABILITY, LOOSE COUPLING 22 LOOSE COUPLING Mediators Compensation Control of process logic Versioning

23 ETL 23 EXTRACT, TRANSFORM, LOAD We need information to support decisions 70% of costs to support decision making We can identify: Effectivity Most important customers Department of highest returns Activities to lower expenses

24 ETL 24 EXTRACT, TRANSFORM, LOAD

25 ETL 25 EXTRACT First we have to identify the data we want Most of the time we extract more than we need This must be done periodically

26 ETL 26 METHODOLOGY OF EXTRACTION Should not affect the resource systems Methodologies: Update notification Incremental extraction Full extraction

27 ETL 27 DATA CLEANING Quality criteria: Validity Accuracy Completness Consistency Uniformity

28 ETL 28 DATA CLEANING Process: Data auditing (microsoft access, file maker pro) Workflow specification Workflow execution Post-processing and controlling

29 ETL 29 DATA CLEANING Steps to improve data quality: Declare a high level commitment to a data quality culture Drive process reengineering at the executive level Spend money to improve the data entry environment Spend money to improve application integration Spend money to change how processes work Promote end-to-end team awareness Promote interdepartmental cooperation Publicly celebrate data quality excellence Continuously measure and improve data quality

30 ETC 30 DATA TRANSFORMATION

31 ETL 31 DATA LOADING The load stage of the ETL process is going to be largely dependent on what you intend to do with the data once it is loaded into the host source. This could include: Layering a business intelligence or analytics tool on top of the warehouse Creating a tool for site search Building a machine learning algorithm to detect fraud Implementing a real-time alerting system

32 ETL 32 DATA LOADING Methods: Full load Incremental load: Streaming Batch