EMC PERSPECTIVE. EMC IT Best Practices for Oracle: Enable Reduced Infrastructure Costs, Improved Performance, and Increased Availability

Transcription

1 EMC PERSPECTIVE EMC IT Best Practices for Oracle: Enable Reduced Infrastructure Costs, Improved Performance, and Increased Availability

2 Table of Contents Executive summary...3 Enterprise-wide infrastructure consolidation...4 Economies of scale with global instance...5 Maximizing application uptime in global instance...6 Multi-million dollar cost savings driven by grid computing...7 Cost efficiency and flexibility through virtualization...8 Data warehouse consolidation increases competitive advantage...9 Enterprise Flash drive performance gains...10 Flexible storage tiering and consolidation...11 Application retirement and archiving drive down storage growth...12 Multi-level IT initiatives have significant impact...13 Sharing best practices in the Oracle IT community

3 Executive summary At EMC Corporation s manufacturing facilities in the U.S., Ireland, and Brazil, EMC information storage systems are assembled, software is loaded, and integrated solutions are packaged for distribution 24 hours a day. In a single quarter, EMC processes tens of thousands of orders generated by its 400 sales offices and thousands of partners, and ships products to customers in 60 countries worldwide. This high-throttle, unending cycle of manufacturing, ordering, and shipping generates exponential volumes of data for EMC to manage, store, and protect. While EMC information infrastructure solutions are designed to help customers grow and operate efficiently, EMC shares the same strategic objectives for its own global IT environment. In the last five years, it has been more challenging to meet these goals as EMC has experienced explosive growth of its revenue, product line, customer base, and an IT infrastructure spanning five global data centers. EMC s Oracle implementation, which is one of Oracle s largest customer environments in the world and spans 800 databases, has encountered percent data growth annually. With this expansion, there has been significant strain on application performance, scalability, and availability. Rapid data growth has also made it challenging to keep up with increasing demand for data center space and energy resources. EMC was faced with spending $120 million to build a new data center or $26 million for a data center upgrade. EMC IT decided it could manage growth more successfully and cost effectively by deploying an efficient IT infrastructure strategy that included consolidation, virtualization, best-in-class backup and replication, as well as data deduplication. This strategy has provided not only cost reductions but an infrastructure optimized for high data availability and top performance. Last year, EMC publicly committed to significant cost reductions as a way to increase its competitiveness and profitability, says Tony Pagliarulo, EMC s Vice President of Application Development. With IT as one of EMC s fastest growing cost centers, IT efficiency was targeted as a major priority. Our distributed Oracle environment was identified as an especially ripe area for cost reduction. By consolidating our Oracle environments onto the most efficient, green, and scalable infrastructure possible, we ve been able to help make EMC more lean, flexible, and competitive. In fact, Oracle recently recognized EMC s IT efficiency accomplishments with its Enable the Eco-Enterprise award. EMC was cited for significant Oracle database consolidation and VMware server virtualization, saving the company millions of dollars in data center hardware, management, and power and cooling. EMC s newly consolidated Oracle infrastructure is the result of many different enterprise-wide IT initiatives focused on: Improving availability and quality of business-critical data and processes Reducing capital and operational IT expenditures Slowing rampant server and storage growth Increasing IT responsiveness and agility This EMC Perspective highlights several of these projects including: Oracle global single instance Oracle application business continuity Oracle grid computing Virtualization Oracle data warehouse consolidation Business intelligence Enterprise Flash drives Automated tiering Data archiving Application retirement 3

4 Enterprise-wide infrastructure consolidation In the last five years, EMC has grown from an $8.23 billion to a nearly $15 billion company, and acquired more than 40 firms. In addition, the company has continued its rapid transition from a primarily storage business to one with substantially increased focus on software and services. To manage this dramatic transition, EMC was continually adding new custom applications, often each with their own software, storage, and server infrastructure. One of its fastest-growing and most widespread infrastructures supported Oracle, a powerful engine of advanced databases and applications that touches nearly every facet of the business. EMC s 40,000-plus employees access Oracle to manage software development, manufacturing, pricing, sales, ordering, service, financial reporting, E-Business analysis, and other critical functions. Without a unified growth management strategy for Oracle, data and business process redundancy was on the rise, IT capital expenditures and administration costs were increasing, and high performance and availability were becoming more difficult to maintain. To reverse this trend, EMC IT executed a multi-pronged approach to consolidate its Oracle environment. EMC deployed global instances of its Oracle enterprise planning resource (ERP) and E-Business Suite 11i customer relationship management (CRM) environments. The company applied Oracle database grid computing using Oracle Real Application Clusters (RAC). In addition, VMware ESX Server virtualization software was implemented to consolidate physical Oracle servers to more cost-effective virtual machines that leverage EMC s storage and software solutions. Today, EMC s new IT infrastructure supports 170 Oracle application modules. Oracle E-Business Suite 11i and Oracle 11.03, along with Oracle Grid 11g database architectures, utilize the breadth of EMC s industry-leading technology stack of information management solutions and VMware platforms. For optimal efficiency, EMC IT has deployed a tiered storage strategy based on an application rationalization methodology. EMC IT allocates the most critical CRM and ERP production data to its high-end Symmetrix DMX and Symmetrix V-Max storage; file sharing data such as business forms to EMC Celerra networked-attached storage; less critical test and development data to mid-tier EMC CLARiiON networked storage; and backups to EMC Disk Library. The overall EMC storage infrastructure is 6.5 petabytes. EMC uses advanced software, including EMC SRDF, EMC TimeFinder, EMC PowerPath, VMware, and Informatica, to manage and protect the end-to-end lifecycle of its Oracle information infrastructure. 4

5 Economies of scale with global instance EMC adopted Oracle global instance to consolidate multiple Oracle application modules onto a single software platform (see Figure 1). With a global instance of the Oracle E-Business Suite, for example, EMC has 35 applications sharing the same database infrastructure. In addition, EMC operates a separate global instance of its Oracle ERP environment. Figure 1: Global Instance of E-Business Suite key statistics E-Business Suite Key Statistics Very high usage penetration within the enterprise. 50,000+ Named Users 4,000+ Concurrent Users at peak periods 70+ Application Tiers Dell/Linux Two-node RAC Architecture 224 cores Oracle Database 10 R2, Solaris 10 Extremely dense module usage Largest users of Service, Install Base and Contracts, Oracle Sales, Configurator One of the largest users of istore Projects TCA, Order Management, etc. Application Continuity keeping 11 available during maintenance One of the largest deployments of Oracle E-Business Suite in the world Business Statistics Quotes 2,263,231 Orders 674,458 Service Requests 1,391,325 Projects 110,512 IB Instances 59,937,589 Material Transactions 12,305,357 Parties 9,429,011 Relationships 10,356,648 Time Cards 7,336,539 System Statistics DB Size 7TB Number of Rows 8.8 Billion IOPS 15K Interconnect Traffic 3,000 4,000 Blocks/Sec Daily Volume Database Transactions 754/Sec 22 Million/Day Archive Log 1.2 TB Conc. Jobs 70K Workflow Events 250K Workflow Rates 20,193,476 With global instance, EMC has reduced hardware infrastructure costs and eliminated the complexity of managing multiple versions of Oracle software. The result has been improved operational efficiency across dozens of financial, sales, and support processes combined with increased automation and integration of cross-enterprise functions. For example, by standardizing financial processes globally with Oracle Financials, EMC can more quickly establish a complete global view of financial information each quarter. Consequently, EMC has reduced financial close timeframes from days to three-to-four days, providing managers with faster access to financial data, and enabling better-informed planning and decision making. By deploying Oracle Projects, EMC has transitioned from fixed-fee to time-and-materials engagements to improve profit margins of projects handled by its technical services organization. EMC now can more easily and quickly locate and deploy technical support skill sets across the company, which has reduced its dependency on third-party support vendors and increased professional services revenues. When EMC acquires a company, the average time required to integrate the new firm s technical services into EMC s Oracle ERP solution has been slashed from six-to-nine months to less than three months in the global instance environment. This enables EMC to more rapidly sell new services, book revenue, and analyze and track project costs. Global instance also has enabled EMC IT to reduce costs associated with managing Oracle applications and the supporting IT infrastructure. Capital expenditures for new IT assets, such as servers and storage for new Oracle applications, have decreased by 50 percent since the database and IT infrastructure can be easily shared across projects. Oracle licensing fees are lower since the global instance does not have to be replicated to support new projects. Standardized processes for multiple Oracle applications, tighter integration, and a single system to optimize also translate into more efficient and agile software administration. For example, EMC IT can respond more quickly to changing business requirements due to quicker applications development, simplified patching, and consolidated backup and disaster recovery processes. 5

6 Maximizing application uptime in global instance In a global instance environment, the business impact of maintenance-driven application downtime for Oracle E-Business Suite was growing as the number of users increased and their productivity and decision-making were hampered when critical processes were not available. Downtime was caused by frequent maintenance and upgrades that were needed to ensure Oracle E-Business Suite delivered top performance and reliability for EMC s 24x7 global operations. In response, EMC IT developed the Application Continuance Tool (ACT) to enable most production processes to occur during maintenance. Using EMC replication software, EMC IT quickly creates copies of Oracle application environments to facilitate maintenance activities, avoiding the time and complexity of moving data over corporate networks. Primary and replicated production data are stored on an EMC Symmetrix DMX-3 system and EMC Celerra NSX gateway. To facilitate replication and management of this data, EMC uses TimeFinder, EMC Ionix ControlCenter, Celerra Replicator, PowerPath, and SRDF software. Since deploying ACT, EMC has reduced maintenance downtime incidents during a quarter from a total of hours to two-to-three hours. ACT enables users to continue working for 88 percent of the time during off-line operations, resulting in increased satisfaction for EMC internal customers. Since EMC IT has the freedom and flexibility to perform maintenance more often, they can work more efficiently and be more responsive to changing Oracle requirements. The Oracle Applications Users Group (OAUG) awarded EMC IT the Innovator of the year award in 2008 for ACT. 6

7 Multi-million dollar cost savings driven by grid computing Another critical consolidation strategy that produced competitive advantages and savings for EMC is Oracle grid computing. Rather than using servers and storage dedicated to a single Oracle database and associated application modules, EMC manages multiple Oracle databases and applications within a consolidated grid computing architecture of Oracle Real Application Clusters (RACs). Before grid computing, EMC s Oracle 10g database environment comprised 26 applications, nine Oracle software versions, 45 stand-alone database servers, and 51 databases supporting separate development, test, and production environments. The new 10g RAC and grid computing environment supports 32 applications, one Oracle software version, four database servers, and six databases. One 10g RAC supports development and test and another 10g RAC supports production. With Oracle grid computing, EMC has gained more flexibility, scalability, and higher utilization because new applications can be launched into the Oracle RAC cluster environment, leveraging the shared information infrastructure. Provisioning databases for new environments now takes days instead of weeks. In 18 months, EMC has saved $3 million due to cost avoidance associated with Oracle replacement servers and storage, as well as reduced cost savings for IT staff who now manage a single, consistent version of Oracle software. As EMC expands grid computing to other database architectures, such as Oracle 11g, an additional $7 million of savings is expected over the next months. With a consolidated grid architecture, EMC has decreased power usage and floor capacity requirements. In addition, availability and performance have improved across the environment because of standardized and published service levels. 7

8 Cost efficiency and flexibility through virtualization Virtualization is a major initiative to help manage the growth of EMC s Oracle server infrastructure, reduce costs, and increase flexibility. Using VMware ESX Server, EMC IT has virtualized several of its Oracle business intelligence application environments. To date, EMC has 375 virtual machines running Oracle Database, Oracle Fusion Middleware/Weblogic, Hyperion, and Oracle E-Business Suite R12. Being able to divide physical servers into virtual machines has increased overall server utilization and productivity. Over a period of 12 months, EMC IT has calculated that virtualization has reduced costs by 82 percent due to decreased server expenditures, power and cooling resources, rack space, and overall data center space (see Figure 2). Figure 2: Virtualization cost savings Oracle Products (Annual Cost) $1,000,000 $800,000 $600,000 $400,000 $200,000 Oracle Products 0 Virtualized Non-Virtualized Savings EMC IT is planning to implement an additional 216 virtual machines by February With a 1:1 ratio, EMC IT estimates that virtualization will have enabled it to avoid the cost of purchasing and maintaining hundreds of additional physical servers. Because EMC IT can generally provision a new virtual machine in one day, versus a two-to-three month window for a physical server, it is able to deploy new Oracle projects more quickly. The reduction in time to implement a virtual server versus a physical server increases our IT agility and saves months in our project lifecycles, says Pagliarulo. As project delivery timelines shrink because of virtualization and grid computing, we re going to see substantial gains from a business perspective. 8

9 Data warehouse consolidation increases competitive advantage Grid computing and virtualization set the stage for EMC to consolidate three major Oracle data warehouse environments into one. This new data warehouse includes detailed sales data that can be organized by customer type, solution, services, industry, geographic location, timeframe, and other parameters. EMC employees use this data to make critical decisions about product development, pricing, channel strategies, geographic focus, and distribution, and ultimately improve EMC s competitive position. Before the data warehouses were integrated, there was significant data redundancy across EMC s finance, customer service, sales, technical support, and corporate quality organizations. This resulted in increased production processing loads, increased risk of data discrepancies, and inconsistent business rules. It took considerable time for users to retrieve data from different sources, and there was often no way to verify that the data retrieved was the most current. The absence of centralized, readily available data was affecting users ability to analyze information and make decisions. As the data warehouses grew, it became increasingly costly to manage and operate them at consistent service levels. There was significant infrastructure redundancy with multiple servers, storage, and databases often hosting overlapping data. Because the architectures differed, high availability, disaster recovery, and information lifecycle management practices also could not be consistently applied. EMC IT s solution was to rationalize and standardize its business intelligence assets. EMC consolidated three copies of its installed Oracle database to one; four copies of the customer database to one; and four copies of the production database to one. The server environment also was migrated from high-end Sun and Fujistu servers to lower-cost Dell servers running Linux, consolidating from 21 database servers to 11 servers. EMC IT also completely virtualized its Oracle application servers in this environment with VMware software, and migrated the consolidated data warehouse and database servers to the Oracle grid. From a storage perspective, EMC employed the use of a tiered storage strategy, which uses Enterprise Flash drives, Fibre Channel drives and SATA drives on EMC Symmetrix VMAX. This new approach resulted in significant performance increases for critical business intelligence functions including: 10x performance gains for one-day processing 180 percent improvement in batch job performance Data mart loading times reduced by 50 percent 200 percent improvement in dashboard rendering This has given EMC a greater competitive advantage because users can analyze, forecast, and respond more quickly and accurately to rapidly changing business trends and customer requirements. This centralized data warehouse is also more cost effective and easier to manage. Consolidating the data warehouse infrastructure onto the Oracle grid enabled moving to a lower-cost Linux server platform, and decreased storage capacity from to terabytes. In addition, this new centralized approach to EMC s data warehouse has greatly simplified backup and recovery operations. Leveraging EMC TimeFinder, a clone copy of the production data warehouse is taken everyday in the morning. Oracle RMAN database backup is then executed off of this new backup copy instead of production to an EMC Disk Library (EDL). Performing the Oracle backup off of a point-in-time copy of production reduced backup times from 16 hours to 4 hours. With decreased storage capacities, smaller more modular servers, and fewer instances of Oracle to manage, EMC IT has decreased management resources for application support by 30 percent. 9

10 Enterprise Flash drive performance gains Increased performance and transaction loads of the consolidated Oracle data warehouse led to EMC IT s decision to move this environment from Symmetrix DMX to Symmetrix V-Max highperformance Enterprise Flash drives (EFDs). In a cutover lasting only 110 hours, 10 terabytes of data and 1,700 users were migrated to the Symmetrix V-Max environment with Dell Linux servers running Oracle RAC. Compared with Fibre Channel, the EFDs delivered significant performance improvements (see Figure 3): Execution time for near-realtime data reduced from three hours to 56 minutes a 67 percent improvement 62 percent reduction in commissions batch processing time 77 percent reduction in rendering of business Intelligence (OBIEE) dashboards and drilldowns Average 62 percent reduction in processing time for main finance batch jobs Dramatic reductions in feeds to Essbase Figure 3: Selected EFD improvements Selected EFD improvements Overall DS-Core FinFact Margin Cur SMF MV s BookBill GDW IDW In addition, there was a 10:1 consolidation of Fibre Channel drives per EFD. With significantly higher performance and dramatically fewer drives, the EFDs proved to be 88 percent more cost effective than Fibre Channel drives. The EFDs performed Oracle reports replays three times faster than Fibre Channel. To meet our service-level objectives, we need to deliver critical business intelligence to tens of thousands of users, very rapidly and reliably, 24 hours a day, says Ramesh Razdan, EMC s Director, Enterprise Technology Services. Having the right information at the right time is critical to making the right decisions. EMC relies on this information to continually adjust sales, order management, distribution, and other operations to ensure we meet customer and overall market expectations. 10

11 Flexible storage tiering and consolidation EMC IT is leveraging the tiering functionality of Symmetrix V-Max, EMC s storage platform purpose-built for the virtualized environment, to further reduce cost and maximize performance. Oracle database grid information is non-disruptively migrated across multiple tiers of storage, enhancing information lifecycle management (ILM) and data mobility (see Figure 4). Figure 4: Tiered Symmetrix V-Max data warehouse infrastructure EMC IT Consolidated Oracle Optimized Warehouse Using Symmetrix V-Max Primary Site Secondary Site FC, SATA (R19) FC, SATA (R5) NRT Prod REVP PASP DICP NRT Test REVP PAST DICT Flash Drives Objects with most stringent response time Fibre Drives Objects from current & recent few quarters SATA Drives Historical data FC, SATA (R5) Flash Drives EFD (R5) FC, SATA (R19) NRT Dev REVD PASD PRODUCTION Global Data Warehouse Dynamic SRDF BCV/Clone DR Global Data Warehouse FC, SATA (R19) FC, SATA (R5) Backup Copy BCV/Clone BCV/Clone Perf FC, SATA (R19) SATA (R5) Gold Copy SATA (R5) TEST BCV/Clone SATA (R5) DEV EMC Symmetrix V-Max 512 GB Cache 400 GB Flash Drives 300 GB 15K RPM Fibre Drives 1 TB 7200 RPM SATA Drives For example, EMC IT uses Symmetrix V-Max virtual LUN migrations to move data across EFDs, Fibre Channel drives, and SATA drives assigned with varying levels of RAID protection. The virtual LUN migrations have proved to be easy to manage, efficient, and reliable because the target is treated as an internal mirror of the logical device and is assigned its own RAID protection and storage tier. With the flexibility of virtual LUN migrations, Symmetrix V-Max is facilitating EMC s ILM strategies, such as moving entire Oracle databases, table spaces, partitions, or ASM disk groups among storage tiers during production operations. EMC also can more easily adjust service levels and performance requirements according to the criticality of application data. The flexible and automated tiering provided by V-Max makes it a seamless process, states Razdan. We assign our Flash drives to high-transaction data; slightly aged production data to Fibre Channel; and SATA for inactive or less-used data and do it all very efficiently in a single frame. This will help us achieve even more consolidation and cost savings. 11

12 Application retirement and archiving drive down storage growth Data archiving and application retirement was identified as another major opportunity for EMC IT to reduce costs and slow data growth. EMC s Oracle Applications 11i database had been growing at a rate of 160 gigabytes per month with the majority of the growth coming from customer service, direct express, commissions, projects, and contracts. Side effects were slow performance for users and high infrastructure costs as new storage, CPU, and memory were continuously added. Other issues included high maintenance and resource costs for performance tuning, development, storage and Unix administration, and growing backup windows. At this rate, EMC IT forecasted that in three years the 11i database would be 12 terabytes a size that would not be manageable and 3.6 petabytes of storage would be needed in five years. To slow growth, EMC IT is archiving and purging inactive data and reorganizing data through functions such as backup and database compression, as well as decommissioning obsolete or inactive applications. EMC calculates that nearly 300 terabytes of storage is being saved as a result. In the next phase, EMC is focused on reducing the size and growth rate of non-production environments through additional archiving. EMC also is preparing to retire old read-only applications, as well as reclaim storage and hardware used by retired applications, which would improve ERP and CRM performance and reduce maintenance downtime. EMC s analysis shows that these changes would avoid the purchase of 1.2 petabytes of new storage over the next five years based on a conservative projected annual growth rate of 30 percent. As part of this latest phase, EMC IT engaged in a 10-day proof of concept (POC) of an Informatica data archiving and application retirement solution with the assistance of EMC Solutions and Informatica Professional Services. First, the POC focused on archiving the contracts and project accounting modules in the Oracle 11i E-Business Suite, which is projected to save 15 terabytes of storage in the first year and 75 terabytes within three years. The POC also focused on subsetting the process of creating smaller databases from larger, more complex databases. By subsetting EMC s three largest Oracle 11i modules Configurator, Install Base, and Customer EMC IT forecasts it could avoid the purchase of terabytes over five years. The engagement also involved retiring 20 IT read-only applications and archiving retired application data to EMC Centera content-addressed storage. By reclaiming storage hardware, EMC IT expects to eliminate the need to purchase 100 terabytes of storage over five years. 12

13 Multi-level IT initiatives have significant impact Even with these initiatives, Oracle databases will continue to grow as EMC expands into new markets, automates new functions, and accumulates more data with every transaction and process. EMC IT is committed to aggressively evaluating and pursuing the latest technologies and strategies that will help it to continually improve IT efficiency and flexibility, and take cost out of the infrastructure across all facets of its Oracle environment including databases, application modules, servers, storage, and networks. Due to a combination of storage tiering, virtualization, and sophisticated database consolidation, EMC IT will be able to continue driving substantial savings and return on investment from these initiatives. 13

14 Sharing best practices in the Oracle IT community EMC is committed to documenting best practices and sharing lessons learned in the global IT community via its EMC Proven Solutions testing program. One of EMC Proven s most featured application environments is Oracle, which is deployed by thousands of EMC customers worldwide. Many of EMC s Oracle customers regularly access EMC Proven publications to learn about proven strategies that reduce costs and improve IT efficiency. Here are a few examples of recent EMC Proven Oracle white papers: Application Retirement Data Archive Subsetting EMC Corporation Hopkinton Massachusetts In North America EMC 2, EMC, EMC Proven, Celerra, Centera, CLARiiON, ControlCenter, DMX, Ionix, SRDF, Symmetrix, TimeFinder, V-Max, and where information lives are registered trademarks or trademarks of EMC Corporation in the United States and other countries. VMware and ESX are registered trademarks or trademarks of VMware, Inc. in the United States and other jurisdictions. All other trademarks used herein are the property of their respective owners. Copyright 2010 EMC Corporation. All rights reserved. Published in the USA. 01/10 EMC Perspective H6867