Leveraging SOA to construct a CDI Ecosystem Robert Roth

Transcription

1 Leveraging SOA to construct a CDI Ecosystem Robert Roth Director, Integration Architecture Solutions

2 Setting Context About Intuit Bob Roth: Director, IAS Shared Development and group in the CTO organization Responsible for technology directions, roadmaps, decisions, implementation designs, boundaries between architectures and platforms for services in the Enterprise and our Product Offerings Responsible for the team delivering core reference services like Party Reference, Item Reference, Offering Catalog, Subscriptions, others Intuit Inc. Over 7,000 FTEs Over $2.3B revenue for FY ending July 2006, 13% YOY growth over FY05 Major offices in 13 states in US, Offices in Canada and UK Quicken, Quickbooks, TurboTax, Payroll products, Accountants/Pro products plus others Over 25 acquisitions since 1993 just recently completed acquisition of Digital Insight to create Intuit Financial Institutions Division 2

3 Business Need

4 Why Intuit Needed CDI and SOA Numerous acquisitions and rapid company growth created an environment of disintegrated systems and redundant customer data Disparate business policies and rules regarding highly shared data Inconsistent, disintegrated business processes across the numerous business units and functional groups There was quantifiable value to our stakeholders in transforming systems integration across silos Greater E2E Customer Experience across all interactions Growth projections required game-changing strategy & prior integration (P2P, RPC-style) inhibited growth Needed to easily acquire and integrate new companies, new systems and integrate those customer experiences while keeping the trust of our customers in handling their sensitive information 4

5 Why Intuit Needed CDI and SOA (Cont d) Long-term growth in number of customers We re not talking a few million customers here Growing complexity in our relationships with our customers What they license, how they pay, how they use Requires a greater overall customer experience across channels Customer experiences are near-real-time Data flowing thru the enterprise Long-running transactions (orders, licensing) Connecting different functional groups Reliability, Availability, Scalability Failover, Active-Active across data centers Manage ever-changing mix of systems - Agility Easily on-board new apps, external sources Allow Intuit to offer products via multiple usage models 5

6 Approach and Strategy 7

7 Business Architecture Comes First! Started by defining the long-term business and technology objectives What do we ultimately need as a company to manage our customer, partner, EE relationships with integrity? What assumptions do we make about our future offering and market strategies? Defined an architecture that solved for this, balancing long and short-term needs and constraints Business Data Integration Low High Coordination Unification Shared customers, products or suppliers Customers and suppliers may be local or Impact on the other business unit global transactions Globally integrated business processes often Operationally unique business of functions with support of enterprise systems. Autonomous business management Business units with similar or overlapping Business unit control over business process operations. design Centralized management often applying Shared customer/ supplier/ product data functional/ process/ business unity matrices Consensus processes for designing High-level process owners design infrastructure services; IT application standardized processes decisions made in the business units. Centrally mandated databases IT decisions made centrally Diversification Replication Few, if any, shared customers or suppliers Few, if any shared customers Independent transactions Independent transactions aggregated at a Operationally unique business units high level Autonomous business management Operationally similar business units Business unit control over business process Autonomous business unit leaders with design limited discretion over processes Few data standards across business units Federated business process design Most IT decisions made with business units. Standardized data definitions but locally owned with some aggregation at corporate Low High Business Process Standardization Solving for our customers, Intuit, and our shareholders Articulated the long-term directions for an integration architecture and high-level design strategies Worked with business users How can we work together across organizational lines? 7 Architecture aligning business and technology long and short-term term

8 Technical Architecture Requirements Follow Needed to be agile Easy to onboard new systems Needed to de-couple systems Chose message-based integration Needed to scale certain systems and provide near-real-time customer interaction Needed to scale to thousands of transactions per second Wanted to isolate systems Control vendor lock-in Let individual systems specialize Resilient don t change every time an endpoint changes Minimize ripple effects when we swap out endpoint systems XML schemas based on Intuit s business needs Systems know little-to-nothing about each other IAS Platform Adopted Buy what we can build what we must abstract to protect from change 8

9 Approaching the Problem phase 1 Leveraged a huge ERP-replacement project in to make move toward EAI and CDI initiatives Couldn t wait for everyone in the business to agree on all fronts (think smart, move fast) Prospect = lead = candidate for all business units what is the definition of customer? Took an different tact agree on an independent standard Party reference model began using in mid-90 s Harry Smith is the same regardless of what he s bought from us! BUT, he may want different identities or accounts and we must respect that! We separated the Party s Identity from the kinds of relationships they have had with Intuit Historically we had inextricably linked identity with the customer s relationship(s) with Intuit Identity went into Party Reference Relationships went into Stakeholder Interaction Designed around EAI principles of integration thru message management Goal: eliminate P2P integration, enable scalability & agility Work to to first first achieve shared vision on on the the problem not the the solution 9

10 13 Solution

11 The Overall SOA Architecture ERP CRM Marketing PD Sales Support Party Ref Core Service Architectures Item Ref Enterprise Offering Catalog Orchestration Order Entitle- ment Ref Loca- tion Ref Stake- holder Support- Inter- ability action Control Biz Rules Event Enterprise Service Platform Service Management, Monitoring and SLA Hardware Layer (XML appliances) Enterprise Service Bus (Service-Oriented Architecture, Real- Time Messaging infrastructure, BCP Support) Metadata Architecture (Repository) (Business Object Models, Definitions, Standards, Rules, Policies, Methods) Enterprise Operations Infrastructure 11

12 Solution Design Options Analysis What were the possibilities? Federated? Too tightly-coupled and not H.A. Leverage CRM? Didn t scale, Inflexible schematics, etc. Data Warehouse? Not real-time and not H.A.; relies on ETL tight coupling Master Database with everything? Didn t work with COTS packages, limited confidence in ability to scale out? Hub-and-Spoke The Only Rational Choice for our needs! Endpoint systems can be up or down fast or slow! Can focus on scaling just the hub, not every system in the ecosystem! Can enforce a common data schema for XML messages Overall solution had to scale to hundreds of orders/second, and thousands of messages/second this was an OLTP Party architecture Had to be highly available, reliable, resilient, and recoverable 12 Specialization, Separation of of Concerns, Adaptation, Tuning

13 Solution: The Party Reference System A single-point of reference for identifying parties of interest to Intuit Individuals, organizations, households, other named groups of people De-coupled identity from relationship with Intuit (customer, vendor, partner ) Single point of Integration regardless of endpoint system technology Single point of data quality management Manages single enterprise-wide Party ID Manages shared data - not everything! Established Reference Pattern used elsewhere Heavily Leverages SOA Design Practices Synchronous and Asynchronous messages separately scalable Actions on Parties exposed as discrete service interfaces 13

14 Party Ecosystem SO Architecture CRM External Web Sites External Web Sites 3. Easy to add or remove systems Offerings ERP BI 2. Systems Communicate using Async Messages 1. Isolated systems can be up or down (Approx 20 spoke systems) Non-Compliant Legacy Systems Party Reference System Gateways Offerings 4. talk to 5. hide system specifics Gateways External Systems 14

15 PRS Architecture Outside In Only One Way to Use PRS Thru Its All messages use the same standard schema Synchronous Message Bus (JMS) Party Reference System Asynchronous Batch Async services use pub/sub and queueing patterns Admin/ Analytics Interface For batch imports from external sources 15 Decouple Isolate Standardize Scale

16 Inside PRS Key mappings live in spoke systems Synchronous Message Bus (JMS) Standardization Interface & Business Rules Management Asynchronous First Logic Identity Matching Engine Master Data Management Admin/ Analytics Interface Identity Hub Batch Import/ Export Batch Service Oriented at the boundary and within the solution 16

17 Party Ecosystem Architecture Summary It s all about loose-coupling No ETLs into/out of PRS! Systems use a common XML schema Minimize impacts of technology changes expose functionality in a way that hides all underlying technologies PRS itself is a black box to the outside world PRS is modular under the covers Makes scalability easier some parts scale up, some scale out interaction is service based internally divided into synchronous, asynchronous, and batch (flavor of asynch) These have different SLA/QoS requirements The overall solution, while simple, is still complex 17

18 Results and Lessons Learned 30

19 Basic Timeline Evolution to next generation implemented in Started the Socialization process in 2002 Work began Spring of Architecture 2002 defined mid-2002 Started analysis and modeling in late Initiate and TIBCO licensed Fall of Launched in production Sept 2004 We started discussing architecture in

20 Our Results Delivered the Party Reference System to production 9/2004 Processing millions of requests/day with subsecond response times Few glitches in production close to % availability Achieved > 3,000 messages/second first year across the backbone Now looking to more scalability and stability Active/active across data centers in the future > % uptime This is more than a physical implementation challenge! > 150 <.5 sec response time now has evolved to deliver > 450 TPS at the same SLA.. Increased agility is the next phase Event driven orchestrations for interaction with the PRS system and the ability to tailor results (the matching criteria) based upon user context 20

21 Lessons Learned Difficult to get single set of business rules Different functional groups have different requirements Marketing s definition of customer vs the customer s definition of their identity Different endpoint systems have different minimum data requirements Data quality and alignment can make or break an SOA initiative Design in the necessary level of abstraction from the beginning this is very hard to add later (justification to undo what works to do what is right ) Always tie back to business goals and objectives Try to highlight legacy rules that should be questioned/changed and isolate true business requirements 21

22 Lessons Learned Governance is essential Depending on corporate culture, you may have to wait to build consensus This can take forever Decide who has to agree and who does not (shared vision vs consensus outcome ) Not every group can get exactly what they want lots of legacy processes need to be addressed If a system cannot play, add them later but factor their needs into the design Architect and design for the long-term, implement for the shortterm Everyone must be willing to agree on what s most important for the company s stakeholders you will have to make some tradeoffs You need an ongoing board to govern the myriad tradeoffs and they must be empowered to make their decisions stick! Getting XML to be reusable, flexible, and support SOA principles is hard! It really requires specialist skills and thinking 22

23 Questions? Thank you! 38