White Paper. cmore/connect for SAP Technical Overview. February Contact Person Laura Zachary Product Management

Transcription

1 White Paper February 2014 cmore/connect for SAP Technical Overview Contact Person Laura Zachary Product Management

2 CMORE/CONNECT TECHNICAL OVERVIEW Table of Contents 1. Overview General Architecture ETL Process Creation and Maintenance User Interface SAP Connectivity Theobald XtractIS Enhancements for cmore/connect ETL Process Automation Logging & Monitoring Business Content Cubes Database Tables SSIS Packages About Document: cmoreconnecttechpaper_en.docx 2/13

3 1. Overview 1.1 General cmore/connect is a sophisticated software application that addresses a common problem in data warehouse environments - efficient and reliable connectivity to source systems - especially SAP. The goal of cmore/connect is twofold: first to reduce implementation effort and second to enhance the capabilities of the Microsoft SQL Server BI platform when it comes to designing and maintaining connectivity to source systems. 1.2 Architecture The architecture cmore/connect follows had been tested and proven in projects involving several different transactional data sources including SAP, Microsoft Dynamics NAV, Tagetik and others. The architecture includes a staging area, which serves as a single destination for data from the source systems. This initial step brings the data into a central SQL Server environment that acts as a single point of fact for the data warehouse environment. 2. ETL Process Creation and Maintenance One of the biggest benefits of using cmore/connect is the ability to automatically generate ETL packages based on a specific SAP system. This ensures that the generated packages contain information that is available in a particular SAP environment, as well as any custom fields or data sources that may be implemented in that environment. cmore/connect is able to scan a SAP system and generates ETL packages based on the metadata in the system. These automatically generated packages have been designed using a methodology based on best practices and many years of Document: cmoreconnecttechpaper_en.docx 3/13

4 experience in ETL projects. Packages include specific details for a particular data source, such as the key condition for an extractor that accesses SAP General Ledger data. SAP Metadata in cmore/connect The packages are used to move data from the SAP source system to the staging area. The methodology behind cmore/connect also applies to the creation of staging tables to house the SAP data. By building packages using cmore/connect, it is possible to apply a standard methodology to ETL for your SAP system. cmore/connect leverages SQL Server functionality like package configurations in Integration Services to streamline package management, which is especially helpful in environments where many packages are utilized. Generated packages automatically are configured to use settings that are easily managed graphically in the cmore/connect user interface. This process had been developed specifically to make it easier to manage and maintain configuration information centrally in an intuitive way. 3. User Interface The cmore/connect user interface serves as the main environment for administrators and developers who are involved in the process of data extraction to work in. It simplifies ETL by acting as a centralized, specialized environment that is dedicated to managing and configuring ETL processes that stage SAP data in Microsoft SQL Server. In the cmore/connect UI, you can: Scan and view SAP metadata Maintain configuration and logging environments. This information is stored in the Connect database. Multiple environments (e.g. Production, QA, Development ) can be managed in a single project or in multiple projects. Document: cmoreconnecttechpaper_en.docx 4/13

5 Create and maintain Staging Databases. A Staging Databases is configured for each source system. Manage global configuration information. This reduces maintenance work by storing common settings centrally. Manage environment- specific configuration information. This provides the flexibility to configure individual staging environments separately. Manage data source- specific configuration information. This provides the flexibility to configure individual data sources separately. For example, transactional sources can be set to always run a full load, while transactional sources process only deltas. Configure and run jobs that load SAP data into the Staging Database Set filters to process specific SAP data, for example filtering by Fiscal Period Monitor ETL processes using reports Configuration Settings in the cmore/connect User Interface The UI is built as a Winforms- based.net application. Document: cmoreconnecttechpaper_en.docx 5/13

6 4. SAP Connectivity 4.1 Theobald XtractIS cmore/connect includes a special edition of XtractIS, which has been developed by Theobald software. XtractIS provides connectivity to SAP, and is based on SAP- certified technology. This section describes the differences between XtractIS for cmore/connect and Theobald s Ultimate Edition of XtractIS, which is the most powerful edition that can be purchased directly from Theobald Software. 4.2 Enhancements for cmore/connect The version of XtractIS that ships with cmore/connect includes all of the functionality of XtractIS Ultimate, plus additional enhancements. The list below contains information about these additional features and how they make XtractIS for cmore/connect an excellent choice for supporting ETL processes in an enterprise environment. The following enhancements are included in the DeltaQ Module of XtractIS for cmore/connect. Dynamic Configuration All of the options in the settings menu are configurable with variables. That means that even after deploying a package the settings can be modified simply by changing variable values in the cmore/connect UI. This is a big improvement because without this functionality, a re- deployment process is required after modifications have been made to SSIS packages. The following parameters can be configured dynamically: o Debug details o RFC server tracing o Check Idoc request state o Don t resend trfc errors automatically o Automatic synchronization o Transfer mode DeltaQ Dynamisation Options Document: cmoreconnecttechpaper_en.docx 6/13

7 The features described above can be configured in the Dynamisation area in the DeltaQ SSIS component UI. XtractIS for cmore/connect is the only edition of XtractIS with these options, which are accessed by clicking the Dynamisation link. DeltaQ Settings with Dynamisation Link Configurable Selection Set In addition to the settings listed above, the data selection for an extractor can be configured using an object variable. This feature that makes it possible to set parameters that filter data that is returned by an extractor, based on specific field values such as filtering General Ledger data, based on a specific range of fiscal periods. Without this additional functionality, it would be possible to define selection sets, but only within the SSIS package itself, which requires the package to be re- deployed in order for changes to take effect. With cmore/connect it is possible to define these settings outside the package using the cmore/connect UI. Using this feature reduces maintenance time since it eliminates the need to manually change and re- deploy the package in order to change the behavior of the ETL process. Document: cmoreconnecttechpaper_en.docx 7/13

8 Dynamically Setting Filters in the cmore/connect User Interface Parallel Data Access to a RFC Destination With the standard edition of XtractIS DeltaQ only one connection to a SAP System can be established at a time with DeltaQ. If more than one connection is established, packages may conflict and cause instability in the data transfer between SAP and the staging database. It is possible to avoid this issue by creating multiple RFC destinations and running the SSIS packages against different destinations. This approach is inflexible because a new queue needs to be created on the SAP side for each RFC- Destination/data source combination. It is not possible to shift a data source from one destination to another without re- initializing the Delta Queue. Parallel data access to a single RFC Destination has been implemented in XtractIS for cmore/connect to overcome this limitation. This functionality makes it possible to optimize data transfer by running several SSIS packages in parallel against a single SAP system. Document: cmoreconnecttechpaper_en.docx 8/13

9 5. ETL Process Automation One of the key aspects of cmore/connect is the ability to automate important aspects of the ETL process. Generated SSIS packages are designed based on best practices and experience from large enterprise- scale data warehouse projects. These packages also include support for management, monitoring, and logging with XtractIS for cmore/connect. ETL processes are managed using Jobs in the cmore/connect user interface. These jobs are standard SQL Server Agent jobs, and can be executed using the SQL Server tools as well as from within the cmore/connect user interface. Each job has one or more job steps. A job step represents an ETL process with a SSIS package that has been generated for a SAP data source. This allows data to be processed for multiple data sources manually or on a scheduled basis, depending on business requirements. Job Steps in the cmore/connect User Interface cmore/connect automatically sets up a Master Package to allow for the easy execution of ETL processes for multiple data sources. 6. Logging & Monitoring cmore/connect includes sophisticated logging and monitoring capabilities. cmore/connect is shipped with several reports for monitoring data transfers within the system. The reports display information about the execution of ETL processes for individual SSIS packages and SQL Agent jobs. Reports in the cmore/connect User Interface Document: cmoreconnecttechpaper_en.docx 9/13

10 The reports display statistics such as status and duration of the ETL process for each data source. The reports also make it possible to drill down into detailed logging details about the communication with SAP. These reports are available from within the cmore/connect user interface and can also be published on a website or in a SharePoint environment. Monitoring and logging information is stored centrally in the cmore/connect database. 7. Business Content Business Content is available as an optional extension of the functionality in cmore/connect. The Business Content is intended to speed up development of a Business Intelligence system based on the combination of Microsoft SQL Server and SAP data. Using the Business Content helps to reduce the time and effort spent during a project by using pre- built SSIS packages, database tables, and cubes. The content includes important know- how about the data sources in the source system, such as which fields are used to create meaningful relationships between dimension and fact data. The cubes that are included combine knowledge of SAP data sources in a cmore/connect- built Staging Area with best practices for multidimensional design. Business Content is based on standard data sources that are shipped with a SAP system and can be applied as is on a standard SAP system. Further configuration and customization is possible, enabling the Business Content to support changes in in SAP systems that have been customized, or to meet a specific business need. Business Content is available for the following SAP modules: (FI) Financial Accounting (Accounts Payable, Receivables, General Ledger Accounting) (CO) Profit Center Accounting (CO) Controlling (Cost Objects, Cost Center Acc.) (MM) Inventory Management (MM) Purchasing (PM) Plant Management (PS) Project System 7.1 Cubes Business Content includes multidimensional cubes based on data from standard SAP Data Sources. The cubes reflect a standard view of SAP data that can be tailored to a company s specific business needs. The cubes are built entirely on SSAS and other SQL Server technologies and can be easily customized using the Microsoft BI toolset. Query, reporting, and analysis tools, including Excel, SharePoint, and many others, are able to consume data from these cubes. Cubes can also be diagrammed in a MindMap using onemind. OneMind is a pmone product that includes intuitive, graphical cube design and development functionality and mind- mapping techniques for visualization. Document: cmoreconnecttechpaper_en.docx 10/13

11 7.2 Database Tables Financial Accounting model in OneMind Business content also includes SQL Server database tables. The schema for the tables was designed based on best practices from practical experience in large data warehouse projects. These tables can be used to build a new data warehouse or augment an existing one. The tables include fields with friendly names for easier use from query, reporting, and analysis tools, and they act as a source for the data that is made available via cubes. 7.3 SSIS Packages SSIS packages are included with the Business Content. These SSIS packages map the data from a cmore/connect Staging Area to the database tables. They include business logic that is specific to SAP data sources and how they need to be processed into the Business Content tables for consumption by cubes and other tools. These packages have been developed based on a standard Document: cmoreconnecttechpaper_en.docx 11/13

12 design and best practices from experience developing data warehouses based on SAP data. They may be used as is or customized to add additional, company- specific business logic and data. Document: cmoreconnecttechpaper_en.docx 12/13

13 About Founded in 2007, pmone AG is a software vendor and consultancy specializing in solutions for Business Intelligence and Big Data. pmone builds solutions using the technology platforms of Microsoft and SAP combined with its own software cmore. cmore helps business users quickly build and efficiently operate scalable reporting and analysis solutions that they can extend to meet their changing needs. pmone also sells and implements Tagetik, a leading global software solution for enterprise planning and consolidation. MindBusiness GmbH, a company specializing in SharePoint solutions and services for Office rollouts, is also a member of the pmone Group. pmone has 200 employees in 8 offices in Germany, Austria and Switzerland. pmone AG Freisinger Straße 9 D Unterschleißheim germany@pmone.de Document: cmoreconnecttechpaper_en.docx 13/13