What is Informatica B2B Data Exchange? Solution Architecture

Transcription

1

2 Eccella is a Data Management and Analytics consulting company, transforming the way companies operate and make decisions. We use offthe-shelf software to purpose build solutions for our customers. Our go-to company for our customers Data Exchange needs is Informatica. But what makes us experts on Informatica Data Exchange (DX)? Our leadership hails from Informatica and has been developing, selling, implementing and advocating this solution for over 15 years. We have been implementing this solution successfully, both under Informatica Professional Services as well as the Eccella brand. We are the authors of the official Informatica training material as well as the sole Authorized Training Partner. You will not find a more knowledgeable DX implementation partner than Eccella in the Informatica ecosystem. Since we ve seen this solution outperform anything else in the market and feel so strongly about it, we felt obligated to share some of our insights with you so that you can make a more informed decision when considering Informatica s Data Exchange. This is not a pitch for DX, but our honest observation of what really matters for organizations considering it.

3 What is Informatica B2B Data Exchange? Business-to-Business Data Exchange is an Informatica product suite that truly is a complete trading partner solution for integration: Between any internal application or system to any external business partner system Between any internal application or system to any other internal application or system Across virtually every communication method In virtually any format or standard including unstructured data The any-to-any connectivity and transformation is what separates Informatica from the competition. More than just enabling integration and connectivity, B2B Data Exchange serves as a platform of reusable applications and partner interactions, enabling business users to quickly onboard new partners, leveraging prebuilt modules and to manage, track and monitor the status of each process execution in great detail. Solution Architecture If you consider the fundamental functionality of DX, it can be broken down into the following: 1. Trading Partner Configuration 2. Internal Integration Point Management 3. Transfer of raw data from one point to another through a set of transformations 4. Event tracking There s more advanced functionality related to archiving, reconciliation of messages, scheduling, and more. This functionality is not reflected in the basic architecture of the product Rather than devising a closed, black box system, Informatica harnessed the power of the tools at its disposal and combined them together to provide an exciting Data Exchange offering.

4 PowerCenter With its flagship data workflow management product, PowerCenter, Informatica managed to introduce the power of system and application connectivity to DX. Leveraging PowerCenter, DX users can connect to any system on-prem or in the cloud. Moreover, loyal users of the highly reliable PowerCenter will be glad to know that the entire data processing in DX is being done by utilizing reusable PowerCenter workflows both in batch and realtime. This is great news in terms of being able to leverage workflows organizations already developed, or finding skills in the marketplace, which is swarming with capable PowerCenter developers. Unlike internal applications and systems, connectivity to external parties is done via the Managed File Transfer (MFT) option. MFT support every protocol and transport layer under the sun. FTP, sftp, HTTP, HTTPS, AS1, AS2, and many, many more. The connectivity details are parameterized so that DX can connect to each trading partner system with its own respective credentials.

5 When dealing with exchange of information between internal systems in the organizations, especially via the use of file exchanges, there s no avoiding data transformations. The Informatica Data Exchange suite comes bundled with Data Transformation (DT), one of the strongest, most robust transformation tools in the market today. Data Transformation can perform any-to-any translations of unstructured, semi-structured and complex structured data - think Word and Excel documents, PDFs and industry standards like HL7, SWIFT, EDI-X12, XML and JSON amongst others. With its ability to execute and track activities against each partner and data exchanged using events tracking mechanism in near real time, the Data Exchange (DX) server is the heart of the solution. The engine marries all critical parts- the trading partner configuration, end point management, data workflow and event tracking into one. The server is configured, managed and monitored using an operational front end. It is integrated into PowerCenter for workflow management and execution, but is also able to leverage any core functionality of PowerCenter such as connecting to any possible data storage. In many situations companies leverage additional Informatica tools in their data exchange deployment. The most popular integrated add-on, Data Quality (DQ), allows cleansing of records in real time for a more quality assured data exchange. Data Quality can help validate the correctness of data, fix sent and received data, standardize addresses, deduplicate records, score the quality of data and many other quality checks and actions to assure the high viability of the data.

6 A deeper look In the diagram above you can see the internal design of the Data Exchange solution. If you were to follow the basic implementation recipe, you would need to follow the below steps: 1. Define reusable workflows for each integration process. These workflows are made up of reusable modules that can be turned on or off depending on the file being processed through them. 2. Define integration connections with internal databases and applications. 3. Define a model for the metadata we plan to store for each partner we will be exchanging data with. The metadata can be used in processing or it can define which transformation modules to execute or skip over. 4. On-board internal and external partners. This includes setting them up in the Data Exchange server, setting up integration points, entering their processing metadata, and defining a routing entry for which workflow they will be routed to. We must also develop any custom modules for transformation such as parsing a custom format.

7 In an ideal implementation you will have a minimal set of reusable workflows and a set of reusable integration points with internal and external systems and partners. This then allows business users to onboard new partners with minimal-to-no development by defining requirements in the metadata profile of a partner. This metadata defines the modules to execute and the format of data exchanged. A sample flow of data might be: A file/message is received from a partner via an integration point (any standard communication protocol). The Data Exchange server resolves the partner and associated metadata and then routes the message with processing instructions to a reusable workflow The workflow executes the appropriate custom and reusable modules for format transformation, data quality and ETL Data is then loaded into a database or application, and/or a message/file is returned to the partner While this only the tip of the iceberg, the possibilities for integration with the above model are endless.

8 Core B2B DX Functionality Leveraging the core three components (PowerCenter, Data Transformation & the Data Exchange Server), the DX solution includes: System/Partner Onboarding Business users can leverage pre-built templates to set up connectivity information, routing rules, and Partner Metadata in one central location. Furthermore, DX comes with partner self-service, which can allow partners to onboard themselves and their new file formats for processing. System/Partner Metadata store The metadata store, or profile, is especially significant because custom web UI s can be configured for storing custom metadata for each system or partner. This metadata can be used both within the actual conversion itself, eg. for currency, as well as to define the flow within a process. EXAMPLE: You receive zip files from one partner and tar files from another. You can store these details in the partners profiles and while both files will be forwarded to the same processing workflow, they will also forward profile metadata so the process will know to unzip or tar the files prior during conversion. If a partner switches their file type, a user can simply change the entry in the partner s profile through a web UI and the process will automatically handle the file appropriately without necessitating any development changes. Complete ETL Functionality with Any to Any Transformation As mentioned earlier, PowerCenter provides complete workflow and ETL functionality for traditional data conversion and logic, as well as Data Quality. In addition, Data Transformation may be leveraged to handle unstructured, semi-structured or complex data that is either received or needs to be generated. An extensive set of standards libraries is also available to provide out of the box integration. EXAMPLE: A business receives and sends EDI-X12 messages. Each partner might provide a different version of EDI-X12. The version can be maintained in the partner profile and when messages are received, the workflow will know to parse the appropriate version, and when they are sent to send the appropriate version. If a partner changes versions, a user need only change the version name in the partner profile and the process will automatically convert it appropriately.

9 Extensive Integration The Data Exchange server s Managed File Transfer provides functionality for setting up local listeners for FTP/HTTP and remote access to the full spectrum of communication standards as well as prebuilt access to web portals such as Amazon. In addition, PowerCenter and its PowerExchange extension provide comprehensive integration with relational databases, Mainframe/AS400, nosql databases, Hadoop, and of course most Enterprise Applications such as SAP, Peoplesoft, Siebel, JDEdwards, Hyperion, and many more. End-to-End Tracking, Monitoring and SLA The Data Exchange server also provides built in end-to-end tracking, monitoring and SLA. Every file retrieved or received and every process initiated by schedule or ad hoc creates an event whose lifecycle is tracked and updated as it gets routed and processed. Event statuses are updated and custom monitors can be created to monitor for specific event statuses or scenarios. When a monitor is triggered, an alert or custom process can be initiated. Data Exchange also provides for defining SLAs for each process or file received. Violated SLAs can be tracked and reported on. Exceptions, Ticket System Integration and Reprocessing Upon exception and failure, users can be notified and events can be updated with exception information providing detailed information on what went wrong and how to fix it. Data Exchange can be fully and automatically integrated with ticketing systems as well. A key feature here is the ability to reprocess existing data or a modified version of the data once either the file has been corrected or the logic has been amended. Complex Event Functionality and Delayed Processing Given the complexity of processes, DX provides features for complex events including delaying events for processing, reconciling events for handshaking and acknowledgements, child events and many more. Dashboarding Data Exchange also includes functionality for custom dashboards, which can be embedded into any web application for quick monitoring of jobs and system performance.

10 Archiving Lastly, Data Exchange provides built-in archiving for all files and messages received/retrieved or sent out to ensure a highly performing system and to comply with conventions and legislation. With a better idea of what the solution can do, let's get into a competitive analysis of other currently available options for solving the same problems of enterprise integration.

11 Key Competitors Hand-coding By far the greatest competition to an enterprise-ready Data Exchange system is hand coding. Leveraging Java, perl scripts, shell scripts, SQL, custom drivers and more, custom integrations are developed between each system making it exceedingly difficult to maintain for hundreds of partners. The time to onboard a new system or partner is long and extensive testing cycles are required. High availability, exception handling, monitoring, etc. all need to be built from scratch and 5 years later the developers have left and the code needs to be reverse engineered; these are all standard problems with hand coding. Niche Vendors Different vertical spaces such as ecommerce, healthcare, and financial payment systems have custom applications with out of the box solutions for most of their needs. However, these niche vendor solutions will always be limited to the expected operation they were intended for. As organizations expand and interact with an ever-growing number of applications, systems and new exchange partners, the ability to have a completely flexible and scalable implementation as Data Exchange provides becomes critical; otherwise innovation and growth are stifled. Axway While Axway provides a powerful and compelling B2B solution, it lacks the ETL power and performance of Informatica s PowerCenter as well as the seamless Data Quality functionality and application integration. IBM Sterling IBM s Sterling is generally the strongest competitor to Informatica s Data Exchange solution in that IBM has a broad offering of software, which can be pieced together to provide extensive functionality. Where IBM is lacking is the integration of their products which is rather poor; the performance and functionality of WTX which is slow vs. Data Transformation; and most importantly, the usability and total cost of implementation. The aim of Informatica s Data Exchange solution is to provide an easy and extensible platform that can be implemented quickly and enables use and configuration by business users. IBM s Sterling solution, while functional, has much longer implementation times and is more complex to build. As such it also becomes more difficult to maintain and bring to production.

12 ESB? Questions about ESB are often brought up, however, ESBs are a fundamentally different beast for a Data Exchange solution. While an ESB provides routing and integration between systems and applications within an enterprise, it does not provide access and integration across all communication protocols outside of the enterprise. Moreover, ESB doesn t provide ETL and any-to-any transformation. In fact, ESBs will often be integrated into a Data Exchange solution and the two are complementary. ESBs can add great value for application and system integration within an enterprise, but they do not provide the metadata and configuration driven Exchange solution with a central hub for defining routing, functionality and onboarding. Common Implementation Patterns Use cases for the types of implementations that B2B Data Exchange was built for and where it can provide value: Manufacturing Dozens of internal system exchange in real time and batch for ERP, manufacturing systems, warehouses, hundreds of external business partners customers, vendors, logistics providers, hundreds of different flows in batch and real time, scheduled and asynchronous with multitude of different custom formats Insurance Hundreds of partners for different business functions brokers, adjusters, actuaries, multiple business processes in hundreds of custom formats and locality rules Healthcare Dozens of different types of transactions, hundreds of different external partners hospitals, departments, systems, governments, many different complex message types, reliable real time processing Financial Services Hundreds of different transaction types, thousands of transaction partners, handshaking requirements, different processing and flow for each message type, real time and asynchronous requirements Data Aggregation Millions of documents, thousands of formats with complex parsing/ processing, hundreds of business partners sending data in unique methods ftp, , web crawling, jms, http, etc. all which need to be consumed, parsed, standardized and published as a product.

13 Informatica s B2B DX Value Add Time to market By providing a platform with inbuilt extensive functionality that is easy to use and maintain, the time to market for onboarding a new system/ business partner is greatly reduced. Leveraging modular components and reuse reduces development and test cycles significantly resulting in reliable and high performing processes in less time with less effort. Business users can literally put together processes on their own out of prebuilt modules. Maintainability Having one central hub that leverages modularity and reuse makes maintaining the overall system much easier. Graphical technologies such as PowerCenter, Data Transformation and simple custom UI s as is provided in Data Exchange make it easy to develop and maintain the overall system without logic getting lost in hand coding. Moreover, modules for logic and connectivity are defined once and leveraged throughout the system to ensure consistency and less development overall. Reusability Reusability is at the heart of Data Exchange. You build connections to systems once. Processing workflows once. Modules once. Format conversion once. Then you leverage the partner profile to route data to a process and use the profile metadata to define which process conversion modules are executed on it. This ensures a maintainable and reliable system. Functionality Lastly, the overall functionality of the product enables it to solve virtually any requirement quickly and efficiently. It is truly an any-to-any conversion tool from anyone to anyone with the infrastructure platform for tracking, monitoring, high availability and exception handling that is critical to any organization s platform.

14 Implement Informatica s Data Exchange Data Exchange implementations can vary wildly given the full spectrum of potential applications of the tool, and the scope of a given project. However, there are a number of general principles which may be followed to both ensure success as well as give a clearer picture of the effort required. Data Exchange's greatest strength is that it is founded upon modularity and reuse. Implementations of Data Exchange do not merely create integrations between systems and partners, rather, implementations of Data Exchange provide a platform whereupon integrations can be quickly and easily configured from functional prebuilt modules and minimal custom modules as needed. As such, the initial infrastructure becomes critical in the success of the implementation. Building Blocks of a Data Exchange Solution There are three main components in any Data Exchange implementation: A. Infrastructure - this consists of the model for business partners, modules and design for monitoring and tracking, modules and design for exception handling and reprocessing, and the overall design of the integration points of the various components and systems. This also includes the design for recovery and high availability as well as approach for scaling. B. Processes - these represent a single business process, which may relate to multiple partners and systems. An example might be ingesting patient visit information. In such a flow, visit information may be received from dozens or hundreds of different exchange partners in many different formats, which all need to be loaded into the same application. While there may be custom requirements per source, all sources are part of the same process. C. Modules - these represent the custom individual modules which are executed within a process to implement custom logic e.g. for a given format (inbound or outbound). Modules are custom subcomponents of a process which will contain generic and common logic as well.

15 Approach A multi-phased Agile approach is preferable in that it enables the infrastructure to be proven and process to be delivered early, confirming the design and direction, and providing the "business" with a level of comfort and reliability in the solution. Within each phase we can define 2 week sprints with a specific set of deliverables in each to ensure progress and success. In Phase 1, the infrastructure is developed along with a few (depending on the scale this might be 2-3 or 10-15) medium complexity processes with a limited number of modules for each (2-3). Phase 1 thus proves out the initial design while fleshing out the complexities and achieving a set of quick clear accomplishments. In Phase 2, we choose to simultaneously work on a number of the more (perhaps most) complex processes, as well as to work on adding modules to those processes completed in Phase 1. The more complex processes quickly informs us of any potential changes to infrastructure required early on, and the additional modules on the older processes continue progressing the project forward. The remaining phases continue with simultaneous development of new processes as well as continued work on additional modules for processes developed prior. Each phase should meet project specific milestones for deployment and the availability of resources and timelines will define the number of components in each of these subsequent phases. Implementation Effort From an effort perspective, the initial infrastructure will generally take 4-8 man-weeks to design and implement, depending on the project requirements. Once the infrastructure is complete, processes will generally take 2-3 days for simple processes, 5-7 days for medium complexity processes, and 8-12 days for complex processes. The design will generally minimize the number of processes implemented as the idea is to leverage reusable processes and components. Lastly, modules will take a half day for simple modules, 1-2 days for medium complexity modules and 3-5 days for complex modules. These are generally much quicker to develop and are much greater in number. On top of these estimates, Systems Integration Test and release as well as a factor for rework and changes needs to be taken into consideration. The above general guidelines for scoping effort can be used to define expectations for milestones as well as to define dates based on the complexity and number of processes as well as the available developers.

16 An agile approach as described above also lends itself well to working with business analysts in that quick iterations prove a process or find issues in design and implementation. Once the process is completed, modules may be developed rapidly. As such the greatest bottleneck is developing and confirming functionality of these processes. Building them out with a small number of initial modules becomes the driving component for the overall project. We have often seen success in implementations that were split 50/50 between onshore and offshore. Onshore/onsite specialists focused on developing infrastructure and processes, while offshore was leveraged for implementing the numerous modules to fit into the processes. It is also critical that the implementation partners are chosen with care. Initial infrastructure and processes should be completed by experts with prior experience in the products and similar implementations. Once the initial phase is complete, more general PowerCenter developers (of which there are many) may be leveraged for implementing processes, though quality and experience vs. hourly rate will generally win out overall in terms of time to deliver and overall cost. Constant and consistent communication regarding progress of the effort is imperative and having an agile approach with progress clearly defined at the end of each 2 week sprint as well as overall at the end of each phase ensures the project stays on track and highlights any issues early.

17 Use Case As a practical example, a pharmaceutical Data Exchange implementation took 550 man-days to build a solution for 200 integrations. These were divided into 35 processes and 200 modules which were developed within a 6 month time frame (including testing and change requests). The project began with an architect and a developer to implement the infrastructure, then added 2 developers to work on implementing processes and then ramped up the team to 6 to work on rolling out the substantial number of modules required. In the final month, there were only two developers responsible for maintenance and enablement while testing completed. Other providers, in the healthcare field for example, have also chosen Data Exchange as their complete solution and have implemented solutions for thousands of integrations which were large multi-year projects. The approach enabled them to continuously bring processes live onto the new platform and to take advantage of its features - performance, end to end tracking, monitoring and exception handling, modularity for minimizing maintenance impact and more. Data Exchange implementations span every conceivable business space because it is so central to the operations of any business and organization the exchange of information between systems, applications and organizations is ubiquitous. From finance, to healthcare to government, insurance manufacturing, telecommunications and data aggregators, the ability to exchange any format of data across virtually any means of communication is extremely powerful. For more information, please visit eccellaconsulting.com

18 eccellaconsulting.com US office:+ 1 (855) UK office:+ 44 (0) India office:+ 91 (0)