Spectrum Control Capacity Planning Scott McPeek mcpeek@us.ibm.com Washington Systems Center Bryan Odom odombry@us.ibm.com Software Advanced Technogoly (SWAT) Copyright IBM Corporation 2017.
Capacity Planning with Spectrum Control Agenda Capacity Planning defined; Capacity Planning strategies Fundamental questions that need to be answered Where to Start and Settings to Verify Key Capacity Planning Metrics and Reporting Options Spectrum Control Grouping Options with Alerts Identifying Waste, Inactive and Orphaned Storage Capacity Planning with Spectrum Virtualize and Spectrum Accelerate Putting it all together Copyright IBM Corporation 2017 Accelerate with IBM Storage. 1
Capacity Planning Defined: Capacity planning is the process of determining the production capacity needed by an organization to meet changing demands for its products. The phrase is also used in business computing and information technology as a synonym for capacity management. IT capacity planning involves estimating the storage, computer hardware, software and connection infrastructure resources required over some future period of time. A common concern of enterprises is whether the required resources are in place to handle an increase in users or number of interactions. https://en.wikipedia.org/wiki/capacity_planning Copyright IBM Corporation 2017 Accelerate with IBM Storage. 2
Capacity Planning Strategies The broad classes of capacity planning are lead strategy, lag strategy, match strategy, and adjustment strategy. Lead strategy is adding capacity in anticipation of an increase in demand Lag strategy refers to adding capacity only after the organization is running at full capacity Match strategy is adding capacity in small amounts in response to changing demand in the market Adjustment strategy is adding or reducing capacity in small or large amounts due to consumer's demand, or, due to major changes to product or system architecture. Copyright IBM Corporation 2017 Accelerate with IBM Storage. 3
Capacity Planning Strategies Put in the Storage Perspective Lead strategy is adding capacity in anticipation of an increase in demand This is the whole idea of capacity planning Requires historical metrics, reporting and monitoring to make it happen Lag strategy refers to adding capacity only after the organization is running at full capacity Nobody wants to be in this situation, scrambling to add capacity as quickly as possible Match strategy is adding capacity in small amounts in response to changing demand in the market Depending on the storage hardware, it may be possible to easily add incremental capacity Adjustment strategy is adding or reducing capacity in small or large amounts due to consumer's demand, or, due to major changes to product or system architecture Events outside of business as usual can introduce the need for additional capacity Examples would be a new system replacing an old system or ingesting an acquisition Copyright IBM Corporation 2017 Accelerate with IBM Storage. 4
Capacity Planners Need To Know and Decide on some Fundamental Questions What is the target utilization for storage? Can you track this utilization and be Alerted if projections start to deviate What is the time line for the capacity plan? Is 3 months too short, can we make a 12 month projection with confidence? What is the lead time required to add capacity? As in, if you need more capacity today, how long does it take before new capacity is installed and usable? If a surprise occurs, what can be done to buy time? Can you employ data reduction technologies in an emergency situation? Can you non-disruptively move capacity to alleviate constrained resources? Do performance requirements override capacity needs? Are there strategic directions that need to be kept in mind? Copyright IBM Corporation 2017 Accelerate with IBM Storage. 5
Where to Start? The first thing that must be decided is the metrics to be used for measurement We need to think in terms of useable capacity This means that the concept of Disks or unformatted space may not be the best option Pool Space on the other hand is really formatted space and is all useable capacity To make useable capacity work, all storage should be formatted into pool storage Some customers have hidden capacity as unformatted storage, which really distorts useable space over time In addition, hiding storage in unformatted capacity hinders overall performance The following chart is a good illustration of what happens to normalized metrics when formatted pool storage is returned back to the storage system as unformatted space The charts from now on relate specifically to using Spectrum Control as a Capacity Planning Tool Copyright IBM Corporation 2017 Accelerate with IBM Storage. 6
Pool Capacity versus Allocated, Why not fully allocating storage make analysis difficult Physical Allocation % has a huge change Allocated Space does not really change Pool Capacity decrease caused the Physical Allocation % Increase Copyright IBM Corporation 2017 Accelerate with IBM Storage. 7
Data Reduction Technologies and Capacity Planning While Data Reduction Technology has a huge impact on the amount of data that can be stored, it does not negate the requirement to monitor the environment. Data Reduction Technology includes Thin Provisioning Compression Deduplificaiton Depending on the hardware, data reduction technologies are used in conjunction with each other For the capacity planner, data reduction technology does not matter Capacity utilization will increase, even with data reduction technology in place This capacity must still be monitored and tracked over time Copyright IBM Corporation 2017 Accelerate with IBM Storage. 8
Data Reduction Technologies and Prediction Capacity Planning is predicting future growth based on historical metrics Data Reduction technology adds more variables to growth prediction Will future Thin Provisioning ratios be the same as the past Will Compression ratios hold Will the next application s Deduplication savings be the same as the past Taken together, the Capacity Planner must now not only predict how storage needs are going to grow but also predict the data reduction savings for future data to be stored What we will probably end up with is having to allow for a larger error factor for future capacity needs Copyright IBM Corporation 2017 Accelerate with IBM Storage. 9
First Thing is to verify Capacity History Retention Make sure that your History Retention matches your reporting requirements Capacity History is not expensive in terms of repository storage, so go big Copyright IBM Corporation 2017 Accelerate with IBM Storage. 10
Capacity Planning Metrics A Partial List Storage system capacity metrics Pool capacity metrics Pool cont d Allocated Space (GiB) Assigned Volume Space (GiB) Available Pool Space (GiB) Compression Savings (%) Data Deduplication Savings (%) Data Reduction (%) Physical Allocation (%) Pool Capacity (GiB) Pool Shortfall (%) Raw Disk Capacity (GiB) Total Volume Capacity (GiB) Unallocatable Volume Space (GiB) Unallocated Volume Space (GiB) Unassigned Volume Space (GiB) Virtual Allocation (%) Allocated Space (GiB) Assigned Volume Space (GiB) Available Pool Space (GiB) Available Repository Space (GiB) Available Soft Space (GiB) Capacity (GiB) Compression Savings (%) Data Reduction (%) Enterprise HDD Available Space (GiB) Enterprise HDD Capacity (GiB) Nearline HDD Available Space (GiB) Nearline HDD Capacity (GiB) Physical Allocation (%) Repository Capacity (GiB) Shortfall (%) Soft Space (GiB) SSD Available Space (GiB) SSD Capacity (GiB) Tier 0 Flash Available Space (GiB) Tier 0 Flash Capacity (GiB) Tier 1 Flash Available Space (GiB) Tier 1 Flash Capacity (GiB) Total Volume Capacity (GiB) Unallocatable Volume Space (GiB) Unallocated Volume Space (GiB) Unassigned Volume Space (GiB) Unused Space (GiB) Used Space (GiB) Virtual Allocation (%) The above list is the epitome of TMI: Too Much Information Copyright IBM Corporation 2017 Accelerate with IBM Storage. 11
Spectrum Control Key Capacity Planning Metrics Storage System Metrics Give Big Picture Views You never want to be at capacity at the storage system level Physical Allocation % Shortfall % Pool Metrics Generally Show a Subset of a Storage System Looking at pool consumption helps to identify which applications are causing growth Pool Capacity Allocated Pool Space Available Pool Space Virtual Allocation % Zero Capacity Copyright IBM Corporation 2017 Accelerate with IBM Storage. 12
Key Capacity Metrics to focus on: Physical Allocation % Physical Allocation (%) The percentage of physical capacity in the pools that is allocated to the regular volumes, the thin-provisioned volumes, and the volumes in child pools. Check the value for physical allocation to see: Whether the physical capacity of the pools is fully allocated. That is, the value for physical allocation is 100%. Whether you have sufficient capacity to provision new volumes with storage Whether you have sufficient capacity to allocate to the compressed and thin-provisioned volumes in the pools Available for: All storage systems. Copyright IBM Corporation 2017 Accelerate with IBM Storage. 13
Physical Allocation % for all Block Storage Systems Storage System Physical Allocation % Sorted at the Washington Systems Center This view is of all Storage Systems, in reality we really do not care about them all. Copyright IBM Corporation 2017 Accelerate with IBM Storage. 14
Physical Allocation % for Large Systems General Group Here is a General Group which includes DS8000 s, XIV s and an A9000, sorted on Physical Allocation %. Like most installations, we care about some devices more than others, the DS8870 is getting worrisome. We will talk more about grouping options later in this presentation. Copyright IBM Corporation 2017 Accelerate with IBM Storage. 15
Key Capacity Metrics to focus on: Shortfall % Shortfall (%) The difference between the remaining unallocated volume space and the available capacity of the associated pool, expressed as a percentage of the remaining unallocated volume space. The shortfall represents the relative risk of running out of space for overallocated thin-provisioned volumes. If the pool has sufficient available capacity to satisfy the remaining unallocated volume space, no shortfall exists. As the remaining unallocated volume space grows, or as the available pool capacity decreases, the shortfall increases and the risk of running out of space becomes higher. If the available capacity of the pool is exhausted, the shortfall is 100% and any volumes that are not yet fully allocated have run out of space. If the pool isn't thin-provisioned, the shortfall percentage equals zero. If shortfall percentage isn't calculated for the storage system, the field is left blank. The following formula is used to calculate this value: [(Unallocatable Space Committed but Unallocated Space) 100] You can use this percentage to determine when the amount of over-committed space in a pool is at a critically high level. Specifically, if the physical space in a pool is less than the committed virtual space, then the pool does not have enough space to fulfill the commitment to virtual space. This value represents the percentage of the committed virtual space that is not available in a pool. As more space is used over time by volumes while the pool capacity remains the same, this percentage increases. Copyright IBM Corporation 2017 Accelerate with IBM Storage. 16
Different Levels of Shortfall Storage system Pool Volume Copyright IBM Corporation 2017 Accelerate with IBM Storage.
Shortfall % for all Block Storage Pools This V9000 VersaStack Pool stands out We should interrogate further Copyright IBM Corporation 2017 Accelerate with IBM Storage. 18
Pool Properties Detail to understand Shortfall % 10,649 GiB Pool Capacity 34,776 GiB Volume Capacity 327% Virtual Allocation Copyright IBM Corporation 2017 Accelerate with IBM Storage. 19
Key Capacity Metrics to focus on: Pool Capacity Metrics Pool Capacity (GiB) The total amount of storage space in the pool, which might include overhead space if the disks for the pool aren't formatted. For XIV systems and IBM Spectrum Accelerate, capacity represents the physical or 'hard' capacity of the pool, not the virtual 'soft' capacity. Allocated Pool Space (GiB) The amount of space that is allocated to the regular and thin-provisioned volumes in the pool. If the pool is a parent pool, the amount of space that is allocated to the volumes in the child pools is also included. The space that is allocated for thin-provisioned volumes is less than their virtual capacity, which is shown in the Total Volume Capacity column. If a pool does not contain thin-provisioned volumes, this value is the same as Total Volume Capacity. Allocated space is the same as used space on all storage systems, except for storage systems that run IBM Spectrum Virtualize. These storage systems might have more allocated space than used space if the storage administrator pre-allocated some space for thin-provisioned volumes when the volumes were created. Available Pool Space (GiB) The amount of space that is available to create new volumes in the pool. If the pool is a parent pool, the amount of space that is allocated to the volumes in the child pools is also included. Copyright IBM Corporation 2017 Accelerate with IBM Storage. 20
Pool Capacity, Allocated and Available Space for a V9000 Sorted on Available Pool Space Copyright IBM Corporation 2017 Accelerate with IBM Storage. 21
EasyTier Distributions By Pool Spectrum Control will give capacity information about EasyTier Pools. The decisions to be made about how large to make each tier is complicated by the need for performance. While it might make sense to add more capacity to a lower tier of the EasyTier Pool, the need to lower volume response times might dictate adding more storage to the SSD tier. Copyright IBM Corporation 2017 Accelerate with IBM Storage.
Key Capacity Metrics to focus on: Virtual Allocation % Virtual Allocation (%) The percentage of the physical capacity that is committed to the virtual capacity of the volumes in the pool. If the value exceeds 100%, the physical capacity doesn't meet the demands for virtual capacity. The following formula is used to calculate this value: [(Total Volume Capacity Pool Capacity) 100] This value is available for all pools. Example: If the allocation percentage is 200% for a total storage pool size of 15 GiB, then the virtual capacity that is committed to the volumes in the pool is 30 GiB. This configuration means that twice as much space is committed than is physically contained in the pool. If the allocation percentage is 100% for the same pool, then the virtual capacity that is committed to the pool is 15 GiB. This configuration means that all the physical capacity of the pool is already allocated to volumes. An allocation percentage that is higher than 100% is considered aggressive because insufficient physical capacity is available in the pool to satisfy the maximum allocation for all the thin-provisioned volumes in the pool. In such cases, you can use the value for Shortfall (%) to estimate how critical the shortage of space is for a pool. You can hover the mouse pointer over the percentage bar to view values for the total volume capacity and capacity. Total volume capacity total is the storage space on all of the volumes in the pool. For thin-provisioned volumes, this value includes virtual space. Capacity is the total amount of storage in the pool. Use this metric to track capacity for A9000 Pools Copyright IBM Corporation 2017 Accelerate with IBM Storage. 23
Virtual Allocation % for Pools on an A9000 Since Physical Allocation % is not available on an A9000, we must use Virtual Allocation % to understand Pool utilization Copyright IBM Corporation 2017 Accelerate with IBM Storage. 24
Key Capacity Metrics to focus on: Zero Capacity Zero Capacity: The date, based on space usage trends, when the pool runs out of available space. Depleted is not a good condition for the Capacity Planner Copyright IBM Corporation 2017 Accelerate with IBM Storage. 25
Exporting Capacity Data for Trend Analysis Copyright IBM Corporation 2017 Accelerate with IBM Storage. 26
Trend Analysis with Excel Exporting CSV data to Excel can help visualize growth, but can be cumbersome and probably not practical on a large scale Copyright IBM Corporation 2017 Accelerate with IBM Storage. 27
Cognos Can Also be used for Historical Analysis Cognos Main Menu Pre-Defined Reports Storage Systems Sub Category Next is an example of Storage Systems Historical Capacity Copyright IBM Corporation 2017 Accelerate with IBM Storage. 28
Cognos Can Also be used for Historical Analysis There are a number of pre-defined Cognos reports that show historical analysis, with fixed reporting periods, up to 90 days. Unfortunately, there are no projections. Copyright IBM Corporation 2017 Accelerate with IBM Storage. 29
Exporting Capacity Data via REST API /srm/rest/api/v1/storagesystems/ [ { "Name": "XIV-214-1340004-214", "id": "19379" "Assigned Volume Space": "43", "Available Pool Space": "25,465.90", "Total Volume Capacity": "33,473.74", "Unassigned Volume Space": "19,003.19", "Used Pool Space": "13,862.81", "Used Space": "11,546.79", Another option for custom reporting is using the REST API. The capacity metrics within the Spectrum Control Repository can be exported to create customer reports. Copyright IBM Corporation 2017 Accelerate with IBM Storage.
Spectrum Control Grouping Options Groups allow you to visualize parts of the storage infra-structure to help understand capacity and performance. In addition, you can specify Alerts that are limited to the Group. Applications are really the main form of Groups. You create Filters to specify what is included in the Application, although you can manually add an object if desired. An Application Group can have sub-components. Departments are a way to group Applications to get a larger view of the storage components. Like Applications, Departments can have sub-components. General Groups are different in the sense that you do not use Filters to automatically populate the group. To populate a General Group, you add the storage objects directly, either through selection or multi-select. Tiers are a form of Grouping where you can visualize your storage based on the type of storage. Storage Tiers are generally grouped along the lines of flash storage, spinning disks and SATA devices. Copyright IBM Corporation 2017 Accelerate with IBM Storage. 31
Application Groups Enable Visibility into Storage Consumption View capacity consumption per Application Department Server Hypervisor Copyright IBM Corporation 2017 Accelerate with IBM Storage. 32
The Database Application Group with Related Resources Keep in mind that unless the underlying infrastructure of the related resources are dedicated, then these resources represent shared capacity outside the group. Copyright IBM Corporation 2017 Accelerate with IBM Storage. 33
The Database Application Group Filter based on volume pattern The filter brought in 15 member volumes Copyright IBM Corporation 2017 Accelerate with IBM Storage. 34
The Database Application Group Filter based on volume pattern The Related Resources includes 34 volumes, when you add in all of the copy relationships Copyright IBM Corporation 2017 Accelerate with IBM Storage. 35
Creating applications Ways to populate an application: The servers / hypervisors running the application Enter a volume name pattern that matches your volume naming convention The storage pool that the application runs in Manual selection of volumes Copyright IBM Corporation 2017 Accelerate with IBM Storage.
Email Reports Give A Good Overview of Application Consumption Copyright IBM Corporation 2017 Accelerate with IBM Storage. 37
The Department IBM Apps includes Databases plus more A total of 5 Applications Copyright IBM Corporation 2017 Accelerate with IBM Storage. 38
The Department IBM Apps includes Databases plus more A total of 5 Applications Copyright IBM Corporation 2017 Accelerate with IBM Storage. 39
General Groups are Groups of Storage Objects General Group objects can be storage systems, pools, volumes, disks, pretty much anything. The only caveat is that if you want to apply an Alert, it is at the object level that you created the group with. Copyright IBM Corporation 2017 Accelerate with IBM Storage. 40
General Group Alert at the Storage System Level One Condition If more granularity is needed, then you would create a group containing components of a storage device, like pools or nodes Copyright IBM Corporation 2017 Accelerate with IBM Storage. 41
General Group Alert at the Storage System Level Custom A custom Alert allows you to add conditions together, in this case the Alert is for Physical Allocation and Shortfall Copyright IBM Corporation 2017 Accelerate with IBM Storage. 42
Identifying Waste At The Storage System Level Reclamation looks at inactive and orphaned LUNs to highlight potentially reclaimable storage Copyright IBM Corporation 2017 Accelerate with IBM Storage.
Identifying Waste Drilled Down To The Volume By clicking on the Volumes hyperlink, we can see exactly which volumes are identified as being potentially reclaimable. Copyright IBM Corporation 2017 Accelerate with IBM Storage.
Spectrum Virtualize Family and Capacity Planning Spectrum Virtualize enables some key capabilities for capacity planners that are not available to non-virtualized environments The key feature to exploit with Spectrum Virtualize is non-disruptive data movement Non-disruptive data movement means that free capacity can be moved from one place to another as need be without taking application servers down While you can not move active volumes non-disruptively between clusters, you can move Mdisks between clusters, once removed from a Pool In addition to moving capacity, data reduction techniques can be employed such as compression and thick to thin migrations at the Spectrum Virtualize layer to free up space If a FlashSystem A9000 is being virtualized, then capacity can be moved to storage Pools to take advantage of deduplication, compression and thin provisioning The bottom line is that virtualization gives capacity planners a number of non-disruptive tools that are not found in other environments Copyright IBM Corporation 2017 Accelerate with IBM Storage. 45
Spectrum Accelerate Family and Capacity Planning The Spectrum Accelerate Family of products include the FlashSystem A9000, the XIV Storage System and Spectrum Accelerate software defined block storage. One of the key differences between this family of storage and other types of storage is that the underlying architecture is grid based For the capacity planner, a grid based storage system means that Pools are more of a logical construct rather than a physical construct Since Pools are a logical construct, they become more of a fencing mechanism to help separate storage usage by application If the initial size of a Pool is not set correctly, then it is very easy to increase the size of the Pool by changing the number For this family of storage, we use Virtual Allocation % to measure Pool utilization versus Physical Allocation % on other devices. Copyright IBM Corporation 2017 Accelerate with IBM Storage. 46
Putting It All Together Capacity Planners need to establish targets and understand their timelines Once key metrics are identified, initial focus will be on over all storage system capacity and utilization Secondary to the over all storage system, are the Pools within the storage system where capacity, utilization and growth can be identified at the application level To help visualize storage usage, grouping methodologies can be utilized to understand usage and growth over time Alerts should be utilized at the levels that make sense, either individually or by grouping applications or objects to be notified of changes in capacity and utilization Technologies like Spectrum Virtualize and Spectrum Accelerate can be exploited to give capacity planners more tools to manage capacity and utilization Copyright IBM Corporation 2017 Accelerate with IBM Storage. 47
Accelerate with IBM Storage Webinars The Free IBM Storage Technical Webinar Series Continues in 2017... Washington Systems Center Storage experts cover a variety of technical topics. Audience: Clients who have or are considering acquiring IBM Storage solutions. Business Partners and IBMers are also welcome. How to sign up? Information, schedules, and replays: To automatically receive announcements of upcoming Accelerate with IBM Storage webinars, Clients, Business Partners and IBMers are welcome to send an email request to accelerate-join@hursley.ibm.com. Located in the Accelerate with IBM Storage Blog: https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en 2017 Webinars: Jan 19 - Spectrum Virtualize Updates Feb 2 - DS8000 Configuration & Monitoring using the GUI Mar 9 - Elastic Storage Servers & Spectrum Scale Software Solution Mar 16 - TS7700 Mgmt Classes & Copy Policies Apr 27 - How to Develop a Workload Baseline to Evaluate Flash Storage May 17 - DS8880 Storage Options for IBM i June 21 TS7700 BVIR What You Need to Know June 22 FlashSystem A9000 Update June 29 DS8880 Thin Provisioning July 20 Spectrum Virtualize Encryption July 24 - Advanced uses of IBM Storage/RansomWare attacks August 10 - Accelerate with IBM Storage: Tape, What's New Sept. 7 - Utilizing the BVIR VEHSTATS Reports to Monitor Your TS7700 Environment Sept. 14 DS8880 Technical Update (R8.3) October 3 GDPS Update Copyright IBM Corporation 2017 Accelerate with IBM Storage. October 10 z14 and IO Infrastructure Modernization November 1 - Spectrum Control Capacity Planning November 16 - VersaStack - IBM s On Premise Cloud Solution Register Here: https://ibm2.webex.com/ibm2/onstage/g.php?mtid=e3dad46fed77c9a7f 11cadc4173e2763a November 30 - LTO8 Drive & related updates to IBM tape products Register Here: https://ibm2.webex.com/ibm2/onstage/g.php?mtid=e3b6902270711054 8491a3785652ce61e December 5 - IBM FlashSystem A9000 & A9000R R12.2 Update Register Here: https://ibm2.webex.com/ibm2/onstage/g.php?mtid=e0e88c21e4431592 bb9887d9df8040d80 December 12 - DS8000 DSGUI Live demo Register Here: https://ibm2.webex.com/ibm2/onstage/g.php?mtid=e9cdd72f0e08f40f72 0efefd4699397e9