100 Million Subscriber Performance Test Whitepaper:

Transcription

1 An Oracle White Paper April Million Subscriber Performance Test Whitepaper: Oracle Communications Billing and Revenue Management 7.4 and Oracle Exadata Database Machine X2-8

2 Oracle Communications BRM 7.4 and Oracle Exadata X Million Subscriber Performance Test White Paper Disclaimer The following is intended to outline our general product performance and throughput. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle.

3 Executive Overview... 2 Introduction... 2 Project Goals... 5 Oracle Exadata Database Machine X2-8 Configuration BRM Server Configuration Network Configuration Operating System Configuration BRM Schema Layout Performance Test Results Rating and Discounting Results Billing and Invoicing Results CSR Results Conclusion... 19

4 Executive Overview Given the explosive growth in broadband, mobile and machine-to-machine communications traffic, the need for cost-effective, high-throughput, highly-available, and scalable billing solutions is increasingly apparent. In February 2011, the Oracle Communications Global Business Unit (CGBU) Performance Group conducted a performance test to demonstrate the performance and scalability of Oracle Communications Billing and Revenue Management (BRM) with an Oracle Exadata Database Machine X2-8 for a 100 million subscriber base. In this performance test, in which multiple workloads were tested, Oracle achieved the best ever throughput performance for Oracle Communications Billing and Revenue Management. One hundred million subscribers were modeled in two Oracle Real Application Clusters (RAC) nodes. Throughput rates, normalized under a per database schema basis, proved superior to those achieved in all prior performance tests. Introduction This document presents the results from a large scale Oracle Communications Billing and Revenue Management performance test performed in February 2011 at the Oracle Solutions Center in Menlo Park, California. This performance test was performed on Oracle Communications BRM 7.4 the industry s leading convergent billing and revenue management platform and on the latest Oracle Exadata Database Machine X2-8. The performance test was conducted for a 100 million subscriber base, created using a standard Oracle Communications BRM GSM performance test price plan. The main workloads were: Rating and discounting Billing with deferred taxation Detailed invoicing CSR activities The throughput results reported in the document reflect either the number of operations executed for each workload given in a fixed execution time, or the observed rate of execution 2

5 against a fixed number of subscribers. The 100 million subscribers were evenly distributed into eight BRM schemas. The full rack Oracle Exadata Database Machine X2-8 was running Oracle Enterprise Linux 5 and Oracle Database 11g Release 2. The Oracle Communications BRM tier ran on Oracle s Sun Fire X4270 M2 servers running Oracle Enterprise Linux 5. Results Summary TABLE 1. PERFORMANCE TEST RESULTS SUMMARY RATING AND DISCOUNTING TOTAL SUBSCRIBERS SUBSCRIBERS PER RAC NODE TOTAL THROUGHPUT (CDRS PER SECOND) THROUGHPUT PER SCHEMA (CDRS PER SECOND) 100 million 50 million 50,125 6,266 BILLING TOTAL SUBSCRIBERS SUBSCRIBERS PER RAC NODE TOTAL THROUGHPUT (BILLS PER SECOND) THROUGHPUT PER SCHEMA (BILLS PER SECOND) 100 million 50 million 2, INVOICING TOTAL SUBSCRIBERS SUBSCRIBERS PER RAC NODE TOTAL THROUGHPUT (INVOICES PER SECOND) THROUGHPUT PER SCHEMA (INVOICES PER SECOND) 100 million 50 million 4, CSR ACTIVITY OPERATIONS PER SECOND OPERATIONS PER SCHEMA LATENCY(MS) 41,580 5,

6 Key Takeaways Oracle Communications Billing and Revenue Management 7.4 with Oracle Exadata Database Machine X2-8 together have demonstrated unprecedented performance levels and the ability to satisfy the needs of the most demanding Tier 1 service providers. The performance test results demonstrate that: Oracle Communications BRM 7.4 with Oracle Exadata Database Machine X2-8 is able to process more than 50,000 operations/sec; operators using this technology can process 2 billion CDRs in less than 12 hours. Billing achieved processing speeds of 2,500 subscribers/sec; operators using this technology can complete a bill run for 100 million subscribers in less than 12 hours. Each Oracle Exadata Database Machine RAC node is able to support at least 50 million subscribers. This reflects an approximate 10x reduction in database server CPU requirements. Oracle Communications BRM with Exadata can greatly simplify production deployments at a fraction of the cost of traditional technology. Within the confines of this test, one Oracle Exadata Database Machine X2-8 replaced 48 conventional x86 servers and 10TB of SAN storage. 4

7 Project Goals The primary goal of the performance test was to measure the performance of the industry-leading convergent billing and revenue management platform, Oracle Communications BRM 7.4, on the latest Oracle Exadata Database Machine X2-8 in a multi-schema configuration. The BRM Rated Event Loader (REL) feature was run to measure the rating and loading performance of batched CDRs for GSM types of services. The main performance goals of this performance test were to measure batch rating and discounting, billing and invoicing results under realistic workloads. Another goal was to determine Oracle Communications BRM, Oracle Exadata Database Machine X2-8 and Oracle RAC scalability over a multi-node DB configuration. The results of this performance test provide the technical data necessary to transfer performance capacity planning information to the Oracle field (professional services and sales organizations), system integrators, and partners for Oracle s Sun platform. 5

8 Application and Hardware Overview The following section provides a brief description of the applications and hardware systems that comprised this performance test. Oracle Communications Billing and Revenue Management The Oracle Communications Billing and Revenue Management architecture provides the foundation for the end-to-end Oracle Communications Billing and Revenue Management solution. Functionality is segregated into layers using well-defined interfaces, enabling each to be modified and enhanced without disruption of functionality at the platform level. At the same time, this design enables the platform to evolve without adverse effects on functional capabilities. As a result, Oracle is able to develop and advance Oracle Communications Billing and Revenue Management s capabilities rapidly while enabling service providers to easily extend the system to meet their unique business requirements. Figure 1. Oracle Communications Billing and Revenue Management architecture The Oracle Communications Billing and Revenue Management platform utilizes a modern n-tier architecture, which is typically deployed with the following three-tier configuration: 6

9 The Applications Tier The applications tier consists of programs and processes that use the object-oriented API as an interface to the Oracle Communications Billing and Revenue Management system. This tier includes the Oracle Communications Billing and Revenue Management client applications such as Customer Center, Pricing Center, and other native as well as third-party applications developed by customers and partners using the same documented API. All enterprise as well as network applications can integrate and interact with the Oracle Communications Billing and Revenue Management system through the applications tier. BRM Server Tier Oracle Communications Billing and Revenue Management server tier consists of three components: the Real-Time Revenue Management Server, the Batch Processing Server, and the Object Server. The Real-Time Engine enforces business policies. It consists of Connection Managers (CMs) and discount, rerating, zoning, and facilities modules. The CMs manage connections between the application tier and the functional modules, process data collected by Oracle Communications Billing and Revenue Management client applications, and enforce the business rules. This architecture allows easy customization of the Oracle Communications Billing and Revenue Management system to meet unique business requirements. The Batch Processing Engine is optimized to handle large volumes of transactions (e.g. hundreds of millions of transactions per day) in terms of preprocessing, enrichment, duplicate checking, aggregation, and rating - among other functions. The Object Server provides an abstraction of the stored data. This layer consists of Data Managers (DMs) that translate requests from CMs into a language recognized by the Oracle Communications Billing and Revenue Management database or other data access systems. The Oracle Communications Billing and Revenue Management system provides separate DMs for each supported database. In addition, there are DMs for other external systems, such as the credit card processing service provided by the Payment Managers. The Data Tier The data tier consists of Oracle Database and other data access systems. The Oracle Communications Billing and Revenue Management application currently supports Oracle Database Enterprise Edition as well as Oracle RAC as the primary database system. Oracle Communications Billing and Revenue Management is also pre-integrated with other data access systems in this tier; for example, credit card processing, LDAP and taxation systems. Scalability The Oracle Communications Billing and Revenue Management application achieves high scalability in several ways. First, its multi-tier architecture is designed to run all Billing and Revenue Management processes on the same computer or distributed among several computers. Distributed processing allows for maximum flexibility and optimal load distribution for configuring an Oracle 7

10 Communications Billing and Revenue Management system as the number of users expands. As a result, service providers can add as many servers as required in the application, Oracle Communications Billing and Revenue Management server, and data tiers. Figure 2. Oracle Communications Billing and Revenue Management scalability architecture A second factor that contributes to the high scalability of Oracle Communications Billing and Revenue Management is the combination of transactional real-time and near real-time batch processing that utilizes a multi-threaded, pipelined architecture and in-memory processing. Integrated support for both types of rating makes extremely high performance possible. To further improve scalability, Oracle Communications Billing and Revenue Management lets providers easily balance load processing by reconfiguring client and database connections. This enables providers to smooth out usage spikes at the front-end and avoid bottlenecks at the database level. Finally, transaction management functions have been built into the object layer, enabling Oracle Communications Billing and Revenue Management to fully scale and take advantage of the underlying hardware. Without this capability, load balancing would be limited beyond a certain transaction level. Oracle Exadata Database Machine X2-8 Oracle Exadata Database Machine provides an optimal solution for all database workloads, ranging from scan-intensive data warehouse applications to highly concurrent online transaction processing (OLTP) applications. With its combination of smart Oracle Exadata Storage Server Software, complete and intelligent Oracle Database software, and the latest industry-standard hardware components, Oracle Exadata Database Machine delivers extreme performance in a highly-available, highly-secure environment. Extreme Performance for Online Transaction Processing, Data Warehousing and Consolidating Mixed Workloads 8

11 The Oracle Exadata Database Machine is an easy-to-deploy, out-of-the-box solution for hosting the Oracle Database. By engineering its hardware and software together, Oracle has eliminated much of the integration effort, cost and effort of database deployment. The system can be deployed for OLTP, data warehousing, or mixed application workloads, lets service providers consolidate multiple computing environments in the data center, and delivers unparalleled performance. The unique technology driving the performance advantages of the Exadata Database Machine is the Oracle Exadata Storage Server. By pushing SQL processing to the Exadata Storage Server all the disks can operate in parallel, reducing database server CPU consumption while using much less bandwidth to move data between storage and database servers. As data volumes continue to grow exponentially, conventional storage arrays struggle to efficiently process terabytes of data, and push that data through storage networks to achieve the performance necessary for demanding database applications. Exadata Storage Servers provide a high-bandwidth, massively parallel solution delivering up to 75 GB per second of raw I/O bandwidth and up to 1,500,000 I/O operations per second (IOPS). Much of these performance gains come from the incorporation of Exadata Smart Flash Cache in each Exadata Storage Server and the Oracle Databases storage hierarchy. With 14 Exadata Storage Servers in a 42U Rack, 5.3 TB of Exadata Smart Flash Cache is integrated into theexadata Database Machine X2-8. In addition, Exadata Database Machine is the world's most secure database system. Building on the high security capabilities in every Oracle Database, Exadata Database Machine provides the ability to query fully encrypted databases with virtually no overhead at hundreds of gigabytes per second. This is done by moving decryption processing from software into the Exadata Storage Server hardware. Figure 3. The Oracle Exadata Database Machine X2-8 9

12 Extreme Scalability The Exadata Database Machine X2-8 is a full rack system with 2 database servers and 14 Exadata Storage Servers. Each database server comes with 64 Intel CPU cores (8 x 8-core Intel Xeon X7560 processors) and 1 TB of memory. It is available with either 600 GB High Performance SAS disks or 2 TB High Capacity SAS disks. While an Exadata Database Machine X2-8 rack is an extremely powerful system, a building-block approach is used that allows Exadata Database Machine X2-8 to scale to almost any size. Exadata Database Machine X2-8 racks can be connected using the integrated InfiniBand fabric. As new racks of Exadata Database Machines are incrementally added to a system, the storage capacity and performance of the system grow. A system composed of two Exadata Database Machines X2-8 racks is simply twice as powerful as a single rack system providing double the I/O throughput and double the storage capacity. It can be run in single system image mode or logically partitioned for consolidation of multiple databases. Scaling out is easy with Exadata Database Machine. Oracle RAC can dynamically add more processing power and Automatic Storage Management (ASM) can dynamically rebalance the data across Exadata Storage Servers to fully utilize all the hardware in each configuration. 10

13 Performance Testing Environment This section describes the system architecture and configuration environment for the Oracle Communications Billing and Revenue Management and Oracle Exadata Database Machine X2-8 performance test. Hardware Configuration The system layout is displayed below: BRM Servers Figure 4. System layout for BRM and Exadata X2-8 performance test 11

14 The system layout consisted of: 1 x full rack Oracle Exadata Database Machine X2-8, used as Oracle database and storage servers 17 x Sun Fire X4270 M2 servers used as BRM application servers 2 x Cisco Gigabit Switches Oracle Exadata Database Machine X2-8 Configuration The full rack Oracle Exadata Database Machine X2-8 contained the Exadata Database Server, the Exadata Storage Server, and the InfiniBand Network. The Exadata Database Server consisted of two Sun Fire X4800 servers to form a 2-node Oracle RAC; each server was configured with: 8 x 8-core Intel X GHz, total 64 cores, 128 threads 1 TB RAM 6 x Gigabit Ethernet ports bonded together, connected to the public/performance test network The Exadata Storage Server consisted of 14 Sun Fire X4270 M2 storage servers; each server was configured with: 2 x 6-core Intel L GHz, total 12 cores, 24 threads 24 GB RAM 12 x 600 GB HP (High Performance) disks. 386 GB flash storage There were a total of 168 disks from the 14 storage servers. Three logical devices were carved out of each disk, one for the DBFS ASM Disk Group, one for the DATA ASM Disk Group, one for the RECO ASM Disk Group. This way every disk group was spread across all disks. All three disk groups were created with normal redundancy (one mirror copy). The BRM DB was initially created on the DATA Disk Group. There was a total of 5.3 TB of flash storage across 14 storage servers. A 2 TB Flash Disk Group was created with normal redundancy, leaving 1.3 TB of flash for regular I/O to disks. The Flash Disk Group was used to offload the intensive I/O to BRM DB. BRM Server Configuration Seventeen Sun Fire X4270 M2s were used for the BRM applications. Each server consisted of: 2 x 6-core Intel X GHz; totaling 12 cores, 24 threads 12

15 96 GB RAM 4 x Gigabit Ethernet ports 6 x 300 GB 15K RPM SAS disk drives The 4 x Gigabit Ethernet ports were configured as: 2 x Gigabit Ethernet ports bonded together, connected to the public/performance test network 2 x Gigabit Ethernet ports bonded together, connected to the BRM private network for batch workloads The 6 x 300 GB 15K RPM SAS disk drives were configured as: 1 boot disk 2 disks striped together to hold the BRM software and batch REL interim directory 3 disks striped together to hold the Batch Rating pipeline input/output directory Even though the network and disks on the 17 servers were configured symmetrically, as the servers served different purposes, not all the components were used on each server, e.g., the Batch Rating pipeline input/output directory was used only on the Batch Router server and the eight servers running Batch Rating + REL. The 17 Sun Fire X4270 M2s were configured in the following manner for the BRM workloads - batch, billing, and invoicing: Server 1: Batch Router, Test Driver Server 2-9: 8 x (Batch Rating + REL), 8 x BRM Server Server 10-17: 8 x BRM Server Each BRM workload was evaluated in isolation and the systems were configured accordingly. One instance of BRM Server is defined as 2 CMs, 2 RTPs, 1 DM Oracle, 1 EAI JS, 1 DM IFW SYNC processes. Servers 2-9 each had 1 NFS export from the Batch Rating pipeline input directory. The Batch Router process on Server 1 used these eight NFS mounts to distribute files for the eight Batch Rating instances to process. The NFS ran on the BRM private network for Batch workloads, to minimize network contention and overhead. Network Configuration The 2 RAC nodes from Oracle Exadata Database Machine X2-8 and all 17 app servers were connected via a Gigabit switch to form the public/performance test network. Communication among Batch Router and Batch Rating instances was separated to a private Gigabit network. Jumbo Frame was 13

16 enabled on all NICs and switches. NIC bonding (EtherChannel) was used on the Gibabit NICs to increase network bandwidth. Operating System Configuration All 17 BRM servers were installed with Oracle Enterprise Linux 5 The two RAC nodes were running Oracle Enterprise Linux 5 The 14 storage servers were running Oracle Enterprise Linux 5 BRM Schema Layout The 100 million subscriber base was divided into 8 schemas, 12.5M per schema. The schema size was determined based on physical memory available for one Batch Pipeline instance, startup time of such instance and size of BRM table/index partitions. Since the Oracle Exadata Database Machine X2-8 had two large RAC nodes, each node hosted four schemas. For better connection isolation, eight services were created for the eight schemas so there was a one-to-one mapping. 14

17 Performance Test Results Plan Type The mobile price plan used in this 100 million subscriber performance test is a standard configuration used by Oracle Communications performance engineering. The performance test comprised a mobile price plan that is representative of real world customer cases. The plan comprised two services, a GSM telephony service and an SMS service (internal plan name is BenchTelcoGSMD). The telephony service was a standard voice service with caller ID, call waiting and voice mail. The price plan utilized two resources, Euros and free minutes. It charged 50 Euros per month for telephony and SMS, with 10,000 free minutes for telephony. The GSM tariff model used 15 different impact categories and had 21 rate plan configuration entries. Each call detailed record (CDR) created three balance impacts: two in Euros and one in free minutes. During billing, every account that was billed had two cycle fees and one fold was generated. [A fold is a special event that usually occurs at the end of the accounting cycle. A fold can be configured to either change a resource balance (currency or non-currency) to zero or convert the resource balance into other resources]. In the GSM batch rating tests, performance was measured for and end-to-end system. Each step of the end-to-end system included CDR file routing, rating and discounting, and database updates using REL. The results are expressed in terms of operations per second. For all test cases, runs are only reported when there were no error messages in any log files (operating system, database system, BRM Server) during the performance test run. Billing had the added requirement that the execution against a fixed number of accounts must have successfully completed. Rating and Discounting Results The rating and discounting workload performs evaluations of service usage and associated rating and discounting. This involves processing of CDR files and updating of subscriber balances i.e., end-toend CDR file processing. Every run processed 50 files of 800,000 CDRs, for a total of 40 million CDRs. The more schemas the bigger input files for Router. 15

18 Components Overall CDRs/sec Batch Rating CDRs/sec REL CDRs/sec App1 % Util DB % Util Memory Router, 8 Batch Rating + REL, 8 schemas Router / Batch Rating /0/0 44/6/ Batch Rating /0/0 43/6/ Batch Rating /0/0 44/6/ Batch Rating /0/0 43/6/ Batch Rating /0/0 44/6/ Batch Rating /0/0 43/6/ Batch Rating /0/0 44/6/ Batch Rating /0/0 43/6/ End-to-End (KB) Table 2. Rating and Discounting Results Billing and Invoicing Results Oracle Communications Billing and Revenue Management billing is used to generate a bill for a given subscriber account on a cyclical or on-demand basis. The generation of the bill is dependent on the billing cycle of the account monthly, bi-monthly, quarterly, semi-annually, or annually. To generate a bill for a given customer account, a setup of applications is run that finalizes the bill for a given period. Billing was evaluated as a key workload during this performance test analysis. Following are the billing results: 42 backends per DM_Oracle, 2 DM_Oracle per BRM schema. 88 children for billing, 88 children for detailed invoicing per schema. 100,000 accounts billed/invoiced per schema. 16

19 Workload ops/sec App1 % Util App2 % Util DB1 % Util # events /account Billing w/ Deferred Taxation /11 40/11 32/11/0 100 Billing w/ Deferred Taxation /10 40/11 31/10/0 100 Billing w/ Deferred Taxation /10 40/11 32/11/0 100 Billing w/ Deferred Taxation /11 40/11 31/10/0 100 Billing w/ Deferred Taxation /11 40/11 32/11/0 100 Billing w/ Deferred Taxation /10 40/11 31/10/0 100 Billing w/ Deferred Taxation /10 39/10 32/11/0 100 Billing w/ Deferred Taxation /11 40/11 31/10/0 100 Billing w/ Deferred Taxation Summary Invoice_DETAILED /4 19/5 17/6/0 100 Invoice_DETAILED /4 19/5 14/6/0 100 Invoice_DETAILED /3 19/5 17/6/0 100 Invoice_DETAILED /5 20/5 15/6/0 100 Invoice_DETAILED /4 20/5 15/7/0 100 Invoice_DETAILED /4 18/5 17/6/0 100 Invoice_DETAILED /7 18/8 25/8/0 100 Invoice_DETAILED /9 19/5 26/9/0 100 Invoice_DETAILED Summary Table 3 Detailed Billing and Invoicing Results CSR Results In addition to the rating and discounting, billing and invoicing tests, a standard test call CSR-select was also run to measure the real-time performance of a typical mix of customer service representative (CSR) tasks. Actual BRM administrative tools were traced while performing typical operations of a CSR, and sets of operations were grouped together to represent typical interactions with customers. Detailed distributions of the operations are based on real-world data from our large customers. 17

20 CSR-Select was run with 640 driver threads Test Ops/sec Resp. time DB1 % Util DB2 % Util APP % Util CUSTOMER SERVICE CSR-select ms 19/22/0 11/6/0 12/9 Table 4. CSR Activity Results 18

21 Conclusion The results achieved with Oracle Communications Billing and Revenue Management 7.4 and the Oracle Exadata Database Machine X2-8 performing a 100-million-subscriber workload demonstrate unprecedented performance, scalability and low total cost of ownership. The results demonstrate Oracle Communications Billing and Revenue Management s ability to handle the most stringent workload requirements of Tier 1 service providers while substantially reducing the costs associated with typical deployments. Results demonstrate: Performance of complex rating and discounting of over 50,000 operations/sec, resulting in the ability to process over 2 billion CDRs in a single 12 hour business day. The ability to perform complex billing with taxation for over 2,500 subscribers/sec, the equivalent to processing a bill run for 100 million subscribers in just over 11 hours. An approximate 10x capacity increase in supported subscribers per Oracle RAC node (up to 50 million subscribers) with Oracle Exadata Database Machine. A significant reduction in hardware requirements for Oracle Communications BRM production deployments, as one Oracle Exadata Database Machine replaces the equivalent of 48 conventional x86 servers and 10 TB of external SAN storage (as compared to previous Oracle Communications BRM performance tests). 19

22 Oracle Communications Billing and Revenue Management 7.4 and Oracle Exadata X Million Subscriber Performance Performance Test Whitepaper April 2011 Author: CGBU Performance Group Copyright 2011, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission. Oracle Corporation World Headquarters 500 Oracle Parkway Redwood Shores, CA U.S.A. Worldwide Inquiries: Phone: Fax: Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. UNIX is a registered trademark licensed through X/Open Company, Ltd oracle.com