Catalogic SoftLayer Cookbook- Copy Data Management

Transcription

1 Catalogic SoftLayer Cookbook- Copy Data Management Catalogic Software s ECX Copy Data Management platform, deployed on IBM Cloud s SoftLayer infrastructure and NetApp storage allows clients to manage, orchestrate and analyze Copy Data, in the IBM Cloud, providing full lifecycle management of your Copy Data. ECX provides automated workflows that allow you to streamline the creation and use of data copies for multiple business solutions, such as automated DR and DevOps in the hybrid cloud. Enabling these use cases in the cloud help clients save both CapEx and OpEx. This cookbook describes the setup performed by Catalogic, IBM and NetApp personnel. IBM: Carol Hernandez (carolh@us.ibm.com) NetApp: Craig Thompson (Craig.Thompson@netapp.com) Jacky Ben-Bassat (Jacky.Ben-Bassat@netapp.com) Catalogic Software: Prashant Jagannathan (prashant@catalogicsoftware.com)

2 Summary This document is intended to discuss the specific ways to manage, orchestrate and analyze Copy Data in a hybrid IT infrastructure deployed on IBM Cloud s SoftLayer infrastructure using Catalogic Software s intelligent Copy Data management platform, ECX, and leveraging NetApp Private Storage and VMware vsphere Infrastructure. This document goes through different operations in IT organizations and provides the ability to get control over their Copy Data across the enterprise including the cloud, delivering the right data copy for the right business function, at the right time and in the right location--all within a single, simplified and automated platform. This document covers four major use cases in today s IT environment including Next Generation Data Protection, Automated Disaster Recovery (DR), Test/Dev (or) DevOps and Business Analytics. The document includes details on installation and setup as well as the creation of Copy Data workflows and use data workflows. These workflows are used to spin up instances of application consistent snaps on the SoftLayer cloud infrastructure to be leveraged for multiple use cases including automated DR, DevOps and business analytics. In this document, VMware@SoftLayer refers to the deployment of VMware vsphere infrastructure in a private cloud within the SoftLayer environment. VMware@Home in this case, refers to the deployment of a VMware vsphere on-premises in a Customer environment. 2

3 Table of Contents 1. Copy Data Management & Hybrid Cloud Catalogic Software ECX Overview Hybrid Cloud Test Environment Creating recovery site at SoftLayer Setting up Storage at NetApp Private Storage Setting up VPN connection Installing and Configuring ECX Copy Data Management Use Cases Copy Workflows Use Data Workflows Disaster Recovery Automation Creation of Development/Testing Environment and Business Analytic environments Copy Data Analytics Conclusion

4 1. Copy Data Management & Hybrid Cloud The process of making copies and then utilizing these data copies is a very challenging and time-consuming process. For example, a typical disaster recovery solution involves two sites. These sites are typically in geographically different locations to assure data resiliency. Additionally, in this remote location, there is a mirror image (or a close resemblance thereof) of the physical infrastructure in the primary location. This is done to ensure that the business can meet all of its Service-Level Agreements (SLAs), for each of the applications in the event of a disaster. A more granular breakdown of the steps involved in a DR scenario (as outlined in the VMware@SoftLayer Cookbook DR 2 ) would involve the following operations: Pre-synchronize storage resources between protected and recovery sites Shut down the virtual machines at protected site and prepare them for migration to the recovery site Synchronize storage resources between protected and recovery sites Suspend non-critical virtual machines at recovery site to make room for migrated virtual machines from the protected site Change storage at recovery site to writable Power on the virtual machines at recovery sites (these virtual machines can be powered on in a pre-determined priority order if necessary) Just to test this DR, the process typically takes a full weekend and the process is driven my multiple scripts. Similarly Development and Test environments pose their own challenges. Testing consumes a lot of compute and storage resources and hence cannot be done on production systems. This requires production data to be offloaded to a test setup. Spinning up multiple servers for testing is time consuming and requires considerable amount of storage and compute resources and needs to be built in a fenced off network to avoid conflict with production. A combination of a true Copy Data management solution and the hybrid cloud can help clients tackle challenges of complexity and time-consuming data movement. 2. Catalogic Software ECX Overview Catalogic Software is an independent software firm that delivers the first Copy Data Management platform that works In Place on your existing storage environment, driving operational efficiencies, cost savings and better leverage of your storage assets. In the modern IT environment, copies of production data can be more vital to the business than the production data itself. But with exponential Copy Data growth, and a hodgepodge of existing tools and scripts to manage these copies, IT is often unable to meet the commitments to the business that depend on this data. ECX manages the full lifecycle of Copy Data. ECX allows IT to manage, orchestrate and analyze Copy Data across their enterprise and cloud. ECX allows IT to leverage application consistent data copies to drive use cases like enhanced recovery, automated daily DR, enhanced Test/Dev, DevOps automation and near real-time data access for Business Analytics that allows you to leverage your storage investments to their fullest. ECX Includes an 4

5 actionable catalog that is at the heart of the platform, and layers on top a robust policy engine to manage and orchestrate the Copy Data environment and associated workflows. ECX has a powerful analytics engine with detailed and customizable reports along with an advanced query engine providing deep analysis as well as real-time and historical service-level reporting. ECX is a software solution installs as a virtual machine, requires no agents and deploys within 15 minutes. Catalogic ECX catalogs all of the primary Copy Data (Snapshots, SnapVaults, SnapMirrors ) in your NetApp and VMware environments. This allows you to orchestrate, analyze, search, and report on all your data, allowing you to take full advantage of your data assets. By cataloging all tiers of NetApp Snapshots and VMware objects, search tasks are completed in seconds, enabling you to identify stale or unwanted data, ensure protection compliance through exception-based reporting Additionally, with an intuitive point and click interface, clients can automate and orchestrate Copy Data creation and use data workflows. ECX s automated workflows allow clients to streamline Copy Data management operations for a number of business operations such as automated DR, DevOps and business analytics. 3. Hybrid Cloud Test Environment This document takes into account a typical Customer Hybrid cloud infrastructure running key applications. The setup is as follows: Customer On Premises Environment (VMware@Home) ESX 5.5 4core & 64GB of RAM FAS2540 running cdot 8.2 Virtual Machines hosted on NetApp o Exchange 2010 o Oracle Database server o SQL database server o Active Directory Server Virtual Machines hosted on Local Datastore(Non-NetApp) o Fileserver 1 o Fileserver 2 SoftLayer (VMware@SoftLayer) ESX 5.5 4core & 64GB of RAM NetApp Private Storage (NPS) FAS2540 running cdot 8.2 A diagram of the implementation is shown in Figure 1. Figure 1 Environment Setup 5

6 3.1 Creating recovery site on SoftLayer Our first step is to create a recovery site for our cloud VMware installation on the SoftLayer cloud infrastructure (VMware@SoftLayer). For more information on how to create a VMware single site in SoftLayer, please refer to the vsphere Basic Site Reference Architecture cookbook. There is no need to create a pairing relationship between VMware@Home and VMware@SoftLayer. Virtual Machines will be automatically provisioned by the Use case Workflows as described in Section 4.2 Use Workflows. 3.2 Setting up Storage at NetApp Private Storage Our next Step is to configure the storage system in NetApp Private Storage (NPS) location. For more information on how to configure a NetApp CDOT system in the NetApp Private Storage and SoftLayer, please refer to the link NetApp Private Storage for SoftLayer. SnapMirror relationship and SnapMirror target provisioning will be automatically done and is detailed in Section 4.1 Copy Workflows. 3.3 Setting up VPN connection This step can be accomplished by setting up DNAT and firewall rules for the vcenter instance or a customer-dedicated POP cross connect. However, a VPN tunnel will be used here for added security, lower cost, as well as simplicity. Please refer to Section III (vcloud Connector), Step 4 of the Migrating vsphere Workloads cookbook for more details on how to set up a simple IPSEC VPN connection with a Vyatta gateway appliance. At this point, our sample environment should look like what is depicted in Figure Installing and Configuring ECX ECX can be deployed using a few clicks within a VMware infrastructure. In our setup, ECX is installed in VMware@Home. In vsphere, simply specify the location of the ova file and pick the Host and Network to run the appliance as depicted Figure 2 and the VM is up and running. ECX comes pre-packaged with all of the required software, and once powered on, the console screen points to the link to the portal as seen in Figure 3. 6

7 Figure 2 Deploy ECX Virtual Appliance Figure 3 ECX appliance with link to the Web Portal The portal link can be accessed from any web browser. Figure 4 shows the logon dialog to access the application. Figure 4 ECX Web Portal Login 7

8 The first step is to register the NetApp assets and VMware vcenter resources and capture their respective sites (including the SoftLayer cloud infrastructure). This is a simple and intuitive onetime agentless registration process. To add a new instance, simply right click and select Register or Discover, as shown in Figure 5. Select Discover to automatically identify NetApp storage in a specific IP range or using SNMP as shown in Figure 6. Select Register and fill in the details, as shown in Figure 7. The registration dialog box takes in basic credentials and a Site selection. ECX has a site-based concept to identify resource location and also the capability of selecting a data copy to be used from any of the sites, be it on-premises or using SoftLayer. Figure 5 Right click to Register Resource Figure 6 Auto Discover NetApp resources in IP range or using SNMP 8

9 Figure 7 Register NetApp and VMware Recourses Once the resources are registered, ECX will auto discover high-level object-like storage volumes, the virtual servers and can also be enabled to capture file granularity for network file shares by running the Catalog Policies. Figure 8 shows the Plan page, where all the policies can be created. We will explain them in detail in the following sections Figure 8 ECX Copy Management Plan screen 4. Copy Data Management Use Cases 4.1 Copy Workflows Having visibility into your existing data sets, applications, snaps and mirrors is important to help determine how your data is currently being used today, and to make decisions on how it should be leveraged as you move forward. However, that is only half the battle. The other half starts with the simplification of the existing data copy creation processes through automation, such that the data copies align with the SLAs of the enterprise, while ensuring 9

10 compliance and auditability of the environment via exception-based reporting. As you introduce the creation of data copies in multiple destinations, including the SoftLayer cloud infrastructure, this management and orchestration becomes impossible without the right tools. In this section we will go through steps to create Copy Data workflows that would allow us to take application consistent snaps of Virtual Machines. To demonstrate this we will use VMs hosted on NetApp storage and on Non-NetApp storage. For the NetApp hosted VMs we will orchestrate a VSS snapshot coordinated with a Storage Snapshot, which is by far the most efficient means of gaining an application consistent SnapMirror for this uses case. However, every environment will have some workloads that do not reside on the storage that is providing the replication to the cloud. For this paper we refer to as Non-NetApp storage for this test, ECX orchestrates the creation of VM image copies and replicates them to the NetApp platform through the vcenter API. This is a very efficient and lightweight VM process driven by VMware s VADP utility. Once the VM image copy is captured, ECX can provide all the Cataloging, Orchestration and analytics as it does with native NetApp SnapMirrors. We will also demonstrate the use of key applications like Exchange, MS SQL, Oracle etc., and replicate them to the cloud. Within ECX we will create application consistent Copy Data workflow policies that are aligned with existing best practices, for different data sets, all in one screen. We then set up the ECX Copy Data workflow schedule, retention policies for individual snapshots and the destination, leveraging NetApp Private Storage (NPS) for the SnapMirrors. ECX provides powerful capabilities that simplify the Copy Data creation process. Figure 9 shows how we can select individual VMs or Folders that needs to be copied to the cloud. Figure 9 Source Selection Once VMs are selected, we select the snapshot options. Figure 10 shows how we can select particular VMs for application consistency and also shows how we can specify Snapshot retention and scheduling frequencies. 10

11 Figure 10 Retention Schedule and Application Consistency for VMs on NetApp For Virtual Machines hosted on Non-NetApp storage, in our case FileServer 1 and 2 will require a selection of target destination. The VMs from those datastores will be copied to NetApp volume. For clarity this workflow will orchestrate the creation of VM image copies, replicate them to the NetApp platform and keep them in synch through the vcenter API. This is a very efficient and lightweight VM process driven by VMware s VADP utility. Once the VM image copy is captured, ECX can provide all the Cataloging, Orchestration and analytics as it does with native NetApp SnapMirrors. Figure 11 shows the destination and scheduling options on non- NetApp VMS. In a similar manner, Figure 12 shows how we can select the storage virtual machine (SVM) in the NPS site to replicate the VMs. Figure 11 Destination, Schedule and Application consistency for VMs on heterogeneous storage 11

12 Figure 12 SnapMirror Destination (NetApp Private Storage) Table 1 shows the time it took to create the initial Snapshot and SnapMirror for 5 VMs. It also shows the average time it took to create a Snapshot and SnapMirror copy of the data along with the capacity moved: VM count Total Data Snapshot Time Replication Replication Time (GB) (sec) (GB) (sec) Initial Snapshot Snapshot Snapshot Snapshot Snapshot Snapshot Average (minus initial) Table 1 ECX Copy Data Workflow to NetApp Private Storage for VMs on NetApp Table 2 shows the time it took to create the initial VM Copy and SnapMirror for 2 VMs on non- NetApp storage. It also shows the average time it took to run the VM copy and SnapMirror copy of the data along with the capacity moved: VM Count Total Data (GB) VM Copy Time (sec) Replication (GB) Replication Time (sec) Initial Copy m Snapshot Snapshot Snapshot Snapshot Snapshot Average (minus initial) Table 2 ECX Copy Data Workflow to NetApp Private Storage for non-netapp VMs 12

13 The initial Snapshot took 80 seconds to create and 47 minutes and 52 seconds to replicate into the NetApp Private Storage. Once the initial snap and mirror was created, it took on average 1 min 35sec to take the next set of Snapshots and just over 1 min to create the SnapMirrors in the cloud. This is a true testament to the NetApp Snapshot, SnapMirror and Cluster ONTAP data fabric capabilities enabling quick and easy data movement to the hybrid cloud. 4.2 Use Data Workflows The next step in the process is to create a Use Data workflows. These workflows allow business lines to access the data copies for multiple use cases. ECX Use Data workflows can use a SnapMirror copy to spin up virtual machines in the SoftLayer cloud infrastructure for multiple use cases. In this section we will go through example of setting up daily-automated DR and Spin up a Test/Development Environment Disaster Recovery Automation In this use data workflow we will select 4 VMs to spin up in the SoftLayer cloud infrastructure to test our Disaster Recovery environments. Commonly, dependencies between applications necessitate that they be recovered in a particular order. In a proper DR test for Exchange, we would need to bring up Active Directory first. ECX allows the user to drag and drop the order in which customers would like to have the VMs come up in. This ensures that all of the applications will work properly. Figure 13 shows how we can select the disaster recovery servers and put them in the order we wish them to come up in. In this particular case we will select ADServer, Exchange, SQL and FileServer Figure 13 DR Server Selection & Order Figure 14 DR Location (SoftLayer) 13

14 Figure 14 shows the next step to choose the location where those resources exist. We will select the SoftLayer center located in SanJose, where we would orchestrate these VMs spinning up. Once the location of where these resources will be spun up is selected, we next select the VMware server in the cloud where the applications and data would be available and in which network we would make them available. What lines of business benefit from having these applications available, and what are they going to do with them automated DR testing, DevOps or business analytics? ECX can spin up these resources, leveraging NetApp s FlexClone technology, in a fenced off environment for DR testing, development or analytics. Figure 15 shows the selection of these hosts and network resources. The workflow to test the DR environment can be scheduled to happen every day, as shown in Figure 16. Now that we have setup the policy it will run per schedule and the IT team can come in in the morning to find that the DR environment is successfully running. In Figure 17, we can see the Monitor screen showing successful DR policy run on SoftLayer that spins up VMs every hour and refreshes the copies with the updated data. Figure 15 Host & Network Resource Selection 14

15 Figure 16 Daily Disaster Recovery Testing schedule Figure 17 Time to Spin Up a DR Instance in the Hybrid Cloud Table 3 shows the average time it took in our setup to spin up 5-7 virtual servers in the hybrid cloud to test DR. The average time was 3 minutes and 49 seconds for an average of 242 GB of data. Since the process is instant, the time taken to spin up will be similar for 5 VMs or 100s of VMs. Server Count Total Data (GB) Time to Spin- up (sec) Time for Cleanup (sec) Average Table 3 Time to Spin Up a DR Instance in the SoftLayer 15

16 4.2.2 Creation of Development/Testing Environment and Business Analytic environments Up to this point, this cookbook focused on the applicability of leveraging hybrid cloud for automated DR, but the same analysis could apply to enhanced test/dev or DevOps automation, as well as near real-time data access for business analytic workloads. The assumption is that a client would deploy a hybrid cloud for one use case but would look to leverage the hybrid cloud infrastructure for additional use cases over time. In fact, once a client has access to application consistent SnapMirors in a hybrid cloud, each use case has the same set of requirements for cataloging, reporting, managing and orchestrating the spinning up and spinning down of compute resources for effective data access. This is the ultimate in Copy Data leverage. For enhanced Test/Dev: The same capabilities described in the previous section could be utilized for spinning up Test and Development environments. We can follow the same steps to create test machines in a fenced off environment and use it to test Application queries, Application-based reporting, etc., and not have to provision large amounts of storage assets. Once the test, development or analytics has been completed, we can let ECX tear down the environment, avoiding unnecessary data sprawl (and cost), or to promote the environment to production if that application was going live. Table 4 reflects time to Clean up the Environment. We can automatically tear down the environment when done with testing or we can select Clean Up option in the Job monitor and it takes less than a minute to remove the VMs. This helps to avoid data sprawl and unnecessary costs associated with leaving stale data in the cloud. Server Count Time for Cleanup (sec) Average 39 Table 4 Time to Clean Up DR Instance in SoftLayer For DevOps automation: As software development gets more and more agile, enterprise IT needs to provide the needed data access agility and automation required. The ECX workflows can provide the required orchestration and automation required to leverage enterprise storage in support of these faster development windows. 16

17 The same capabilities described in the previous section could be utilized for spinning up DevOps environments. We can follow the same steps to create test machines in a fenced off environment and use it in Development and not have to provision large amounts of storage assets. Once the test, development has been completed and released, ECX would promote the environment to production. This would seamlessly migrate the environment to production, even as the application is live. Then ECX could immediately create another fenced off environment based upon an application consistent snapshot of the new production environment. This process would continue through a rapid development cycle. In this DevOps model, workflows would be set up by the storage or Dev team but the promoting to production would be most likely controlled by the Development team. All of these workflows can be created and driven by ECX s Rest API. For near-real-time data access for Business Analytics: The same capabilities described in the previous sections could be utilized providing near realtime data access for analytics. We can follow the same steps to create test machines in a fenced off environment and use it to run Business Analytics, Application queries, Application-based reporting, ETL processes, etc., and not have to provision large amounts of storage assets. Once the analytics has been completed, we can let ECX tear down the environment, avoiding unnecessary data sprawl and cost. 4.3 Copy Data Analytics We can also run Analytics on the data collected within ECX catalog. The Report tab provides access to several key Reporting options, as shown in Figure 18, which could be leveraged for day to day activities. We will go through a couple of Reports in particular. Figure 19, shows VMware RPO Compliance, which shows the VMware objects and the relation to underlying storage systems. It also displays the recovery points on-premises in VMware@Home and in the cloud VMware@SoftLayer. Figure 20, shows a VMware VM Sprawl Report. This report is very useful in a Hybrid Cloud environment where compute resources are spun up and not cleaned up after the tests are completed. This report identifies the VMs that are powered off and in a suspended state and shows the storage capacity that can be reclaimed along with their corresponding locations. Figure 18 Reports 17

18 Figure 19 VM RPO Compliance Report Figure 20 VM sprawl Report 5. Conclusion In this cookbook we have demonstrated how to manage, orchestrate and analyze Copy Data in a Hybrid IT deployed on the SoftLayer cloud infrastructure using Catalogic Software s intelligent Copy Data management platform, ECX, and leveraging NetApp Private Storage and VMware vsphere Infrastructure. The document referred to other cookbooks that document detailed set up and testing of important pieces of this test. Additionally, the environment set up and workflow execution was recorded and available in video format at This document covers four major use cases in today s IT environment including Automated 18

19 DR, Test/Dev (or) DevOps and Business Analytics. The document includes details on installation and setup as well as the creation of Copy Data workflows and use data workflows to spin up instances of application consistent snaps onto the SoftLayer cloud infrastructure to be leveraged for multiple use cases including automated DR, DevOps and business analytics. This document did not attempt to create detailed return on investment and/or return on assets calculation. However, the OpEx and CapEx saving of the combined SoftLayer, NetApp and ECX solution over a traditional private cloud seem fairly straightforward and compelling. IT now has a more powerful way to harness the value of IBM Cloud s SoftLayer infrastructure, without adding complexity or requiring added expertise. ECX can automate and orchestrate all of the Copy Data use cases such as Automated DR, enhanced Test/Dev, DevOps automation and near real-time data access for Business Analytics. Trademark Information Catalogic is a registered trademark of Catalogic Software Inc. ECX is a trademark of Catalogic Software Inc. All other company and product names used herein may be the trademarks of their respective companies. NetApp, the NetApp logo, Snaphot, SnapVault, SnapMirror are trademarks or registered trademarks of NetApp, Inc. in the United States, other countries, or both.all other brands or products are trademarks or registered trademarks of their respective holders and should be treated as such. IBM, the IBM logo, ibm.com, PartnerWorld, are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at ibm.com/legal/copytrade.shtml SoftLayer is a registered trademark of SoftLayer, Inc., an IBM Company. Other product, company or service names may be trademarks or service marks of others. All information contained in this document is subject to change without notice. References herein to IBM products and services do not imply that IBM intends to make them available in all countries. The information contained in this document does not affect or change IBM product specifications or warranties. All information contained in this document was obtained in specific environments, and is presented as an illustration. The actual results obtained by any party implementing such products or services will depend on a large number of factors specific to such party s operating environment and may vary significantly. IBM makes no representation that these results can be expected or obtained in any implementation of any such products or services. 19