Efficient IT Operations Require Workload Automation

Transcription

1 A Forrester Consulting Thought Leadership Paper Commissioned By BMC June 2016 Efficient IT Operations Require Workload Automation

2 Table Of Contents Executive Summary... 1 Automation Is Critical For IT Efficiency... 2 Enterprises Challenged By Scripting And Complexity... 3 Enterprise Workload Automation Is Designed For Control... 5 Key Recommendations... 8 Appendix A: Methodology... 9 Appendix B: Demographics... 9 ABOUT FORRESTER CONSULTING Forrester Consulting provides independent and objective research-based consulting to help leaders succeed in their organizations. Ranging in scope from a short strategy session to custom projects, Forrester s Consulting services connect you directly with research analysts who apply expert insight to your specific business challenges. For more information, visit forrester.com/consulting. 2016, Forrester Research, Inc. All rights reserved. Unauthorized reproduction is strictly prohibited. Information is based on best available resources. Opinions reflect judgment at the time and are subject to change. Forrester, Technographics, Forrester Wave, RoleView, TechRadar, and Total Economic Impact are trademarks of Forrester Research, Inc. All other trademarks are the property of their respective companies. For additional information, go to [1-W6N0DO] Project Director: Lisa Smith Market Impact Consultant Contributing Research: Forrester's Infrastructure and Operations research group

3 1 Executive Summary Job scheduling is a necessary component of data centers. Without job scheduling, an enterprise would not be able to execute long application processes that deal with massive amounts of data yet are necessary for online and transactional processing. Distributed systems and complex asynchronous applications have brought new constraints to how enterprises plan, execute, and monitor jobs. Today s IT organization places a stronger emphasis on automation: Ad hoc solutions such as scripting and ad hoc scheduling are no longer up to the task. Over the past several years, we have seen job scheduling evolve into workload automation. Job scheduling started from a discipline that used spare computing time and ad hoc implementations to execute batches. However, it is now an enterprisewide solution for executing asynchronous applications that result from events or user actions. In May 2016, BMC commissioned Forrester Consulting to evaluate how companies are utilizing scripting and job scheduling tools to improve the overall efficiency of IT and support business workflows. Then to further explore this trend, Forrester tested a hypothesis asserting that moving beyond manual scripting and ad hoc scheduling through a controlled and enterprisewide automation of jobs will drive increased IT operational efficiency In conducting in-depth surveys with 180 IT decision-makers, Forrester found that those companies that adopt an enterprisewide automation job scheduler avoid excessive consumption of resources associated with ad hoc scheduling. IT teams face key challenges derived from the complexities of multiple job scheduling tools, including excessive errors resulting in manual rework, failure to complete jobs on time, and difficultly maintaining scripts. However, companies that adopt an enterprisewide workload automation job scheduler benefit from greater control, resulting in an enhanced ability to support the business, a single view of all batch workflows, and the elimination of scripting. KEY FINDINGS Forrester s study yielded four key findings: A majority of enterprises prioritize a more efficient IT department. Sixty-five percent of IT decision-makers recognize that improving IT efficiency is a top business objective for the next year. They view automation and workload automation in particular as a critical solution to reach this objective. IT organizations adopt workload automation to centralize control. Workload automation centralizes execution and provides users with a complete view of all activity. Of the 44% of enterprises that have adopted an enterprisewide scheduler, over half (52%) cited a single view of scheduled jobs as the top driver for adoption. Multiple job schedulers don t bode well for an efficient IT organization. Multiple scheduling approaches result in multiple issues. Top challenges include difficulty maintaining scripts; lack of synchronization between jobs, leading to errors; and inefficient use of resources. Enterprises should let enterprise workload automation govern job scheduling. Shifting away from a manual, ad hoc job scheduling approach toward a centralized, enterprise-level scheduler can eliminate challenges, increase IT productivity, and enhance IT operational efficiency.

4 2 Automation Is Critical For IT Efficiency The majority of enterprises align on business imperatives, with winning, serving, and retaining customers consistently at the top of the list. And as enterprises strive to remain competitive, they demand fine-tuned IT operations to execute upon business needs. However, the current state of IT operations misses the mark. IT organizations have experienced spectacular growth, acquiring diverse IT infrastructure in the process. The result is an IT landscape that is increasingly complex in nature. As such, enterprises are adopting workload automation to improve IT productivity and overall efficiency. This new approach emphasizes agility improvement and the elimination of waste at the operational level in order to ultimately serve business needs better and faster. Our study found: Business initiatives focus on a more efficient IT. The use of well-oiled business IT services has significantly increased business productivity. Organizations strive for similar progress on the IT side. Sixty-five percent of IT decision-makers ranked improving the overall efficiency of IT as the top business-related theme over the next year, and 57% of decision-makers agreed that the use of automation to improve IT flexibility and agility was critical (see Figure 1). An additional 50% prioritized the continued ability of IT to support new business applications. Automation is the key to success. When we asked enterprises how they plan to support slated business initiatives over the next year, they cited the importance of using automation to improve IT agility, and over half (57%) listed plans to implement new automation technologies. These factors go hand in hand in driving IT productivity and industrializing IT operations. In fact, 61% of respondents intend to implement IT process automation technology to reduce manual errors, and 50% plan to deploy workload automation to improve the execution of batch or other tasks. FIGURE 1 Businesses Prioritize Improving The Efficiency Of IT Which of the following initiatives are likely to be your IT organization s major business-or corporaterelated themes in the next 12 months? (Rank your top three) Improving the overall efficiency of IT Using automation to improve IT flexibility and agility Increasing IT s ability to support new business applications Increasing the scope of IT s centralized or shared services Ranked 1 Ranked 2 Ranked 3 Improving the measurement of IT s impact on business performance 10% 14% Defining a strategy for risk 11% 12% 14% and compliance 30% 19% 16% 16% 17% 15% 19% 16% 21% Base: 180 IT decision-makers responsible for workload automation and job scheduling at enterprises within the US, the UK, Germany, and France 65% 57% 50% 47% 46% 37% Adoption of workload automation brings efficiency. As IT organizations strive to establish efficiency, 44% of IT decision-makers turn to an enterprise workload automation solution as their job scheduling software (see Figure 2). Forty-one percent employ a packaged application scheduled solution, and 35% of enterprises still use multiple job schedulers on different platforms.

5 3 FIGURE 2 Forty-Four Percent Of Organizations Currently Use An Enterprise Workload Automation Solution Is your organization using the following job scheduling software? (Select all that apply) An enterprisewide workload automation solution Packaged application scheduling solution In-house-developed solution 39% 44% FIGURE 3 Drivers Of Workload Automation Include Centralization, Standardization, And Cost Efficiency Which of the following items drove your company to move to an enterprisewidescheduler? Providing a view of all scheduled jobs statuses on a single pane of glass Avoiding costs associated with writing and maintaining scripts 46% 52% Multiple job schedulers on different platforms We use scripting to manage batch computing Native platform scheduling solution (e.g., CRON) No, we do not use or plan to use automation Base: 180 IT decision-makers responsible for workload automation and job scheduling at enterprises within the US, the UK, Germany, and France 1% 28% 35% 33% Creating a standard enterprisewide operating model Integrating packaged application schedulers into a single, enterprisewide solution Providing a solution to recover from server and other platform failures Offering development and business groups with a selfservice solution Reducing the business risk of jobs not completed on time 44% 42% 42% 35% 35% Automation provides centralized control. Businesses value centralized control, with more than half (52%) citing the ability to view all scheduled jobs status from a single pane of glass as the top driver for workload automation. In addition, 44% are motivated by a standard enterprisewide operating model, and 46% adopted an enterprise scheduler to avoid costs associated with writing and maintaining scripts (see Figure 3). Decision-makers are evading the expected errors and costs of ad hoc schedulers; they are evaluating solutions that abstract IT complexity and unearth resources. Base: 79 IT decision-makers responsible for workload automation and job scheduling at enterprises within the US, the UK, Germany, and France Enterprises Challenged By Scripting And Complexity While the use of multiple schedulers provides some degree of automation, it also results in unfettered issues: errors requiring manual rework, inefficiency of resources in the form of IT personnel, and lack of synchronization between schedulers. Our study found that: IT decision-makers face challenges with job scheduling. The majority of companies experience challenges with their current job scheduling, including difficulties maintaining scripts over time, excessive manual rework to write and test scripts, and a failure to complete critical jobs in time for the business (see Figure 4). The latter of these challenges can be inordinately expensive for enterprises of all sizes.

6 4 FIGURE 4 Top Challenges Are Maintaining Scripts, Manual Work To Write Scripts, And Failure To Complete Critical Jobs In Time Thinking about workload automation in your organization, how challenging are each of the following issues? (1 to 5 scale where 5 = very challenging ) Difficulties in maintaining scripts over a period of time Excessive manual work to write and test scripts Critical jobs not completed in time for business Having visibility of all jobs being run across the enterprise Lack of synchronization between job runs requiring excessive manual intervention Critical files not being transferred in time for critical job runs Noncritical jobs not completed in time for business Excessive manual work to run jobs on different platforms than the scheduled platform for the job The status of a job in the processing queue % 26% 38% 21% 21% 23% 23% 17% 19% 38% 35% 34% 40% 34% 43% 39% Base: 180 IT decision-makers responsible for workload automation and job scheduling at enterprises within the US, the UK, Germany, and France 64% 64% 60% 59% 58% 57% 57% 53% 52% Disparate job schedulers fuel IT complexity. The dangers associated with native/bundled/free job schedulers are not unfamiliar. When asked about the consequences of multiple schedulers, 65% of IT decisionmakers reported that a lack of integrated and managed file transfer had an impact on operations (see Figure 5). In addition, respondents cited that a lack of synchronization between different job schedulers (62%) and a lack of integration with system management solutions (65%) had a significant impact on teams. Excessive errors hinder IT productivity. The negative consequences of disparate job schedulers unfold in routine IT processes. Sixty-nine percent of companies agreed or strongly agreed that managing multiple unsynchronized jobs is difficult and error prone. The same frequency of companies (69%) cited that a lack of integrated managed file transfers leads to errors and business issues. And finally, 68% of companies said that managing multiple solutions requires excess manual resources, most often in the form of IT personnel. Routine inaccuracies and errors detract from productivity gains and exhaust costly resources. Bundled, free solutions actually carry a high support and business cost. FIGURE 5 Native/Bundled/Free Job Schedulers Have Significant Negative Consequences On a scale of 1 to 5, where 1= little to no impact and 5 = significant impact, what do you believe are the most important consequences of using native/bundled/free job schedulers? Lack of integrated and managed file transfer Lack of integration with system management solutions Difficulties in using a different server than the one initially scheduled Lack of service-level management control to evaluate run times Lack of synchronization between the different job schedulers Lack of synchronization between packaged application schedulers and generic job schedulers % 20% 31% 26% 26% 30% 34% 65% 39% 37% 32% 40% Base: 180 IT decision-makers responsible for workload automation and job scheduling at enterprises within the US, the UK, Germany, and France 65% 65% 63% 62% 60%

7 5 FIGURE 6 Organizations Dedicate Significant Human Resources To Job Scheduling And Script Maintenance How many full-time employees work on job scheduling in your organization? How many full-time employees work on writing and maintaining scripts in your organization? 39% 32% 34% 24% 27% 13% 8% Less than 5 Between 5 and 9 Between 10 and or more Less than 5 Between 5 and 9 Between 10 and or more Base: 180 IT decision-makers responsible for workload automation and job scheduling at enterprises within the US, the UK, Germany, and France ( don t know responses not shown) Multiple job schedulers inflate IT resources. Managing resources with multiple job schedulers is a daunting task. According to our study, a third of organizations dedicate 20 or more full-time employees (FTEs) to job scheduling (see Figure 6). Similarly, 27% have 20 or more FTEs working on, writing, or maintaining scripts in their organization. A lack of holistic automation pushes the burden of managing schedulers and scripts onto IT administrators. batch activities and 59% stated that they encounter reduced manual interventions and errors. Nearly half of respondents (49%) relayed the ability to allocate a number of full-time employees in charge of scheduling to other operational tasks. And it appears that there is still room to delight late adopters according to our study, users of multiple schedulers underestimate the benefits of an enterprise-level solution (see Figure 7). Enterprise Workload Automation Is Designed For Control Modern workload automation presents many advantages over other methods of scheduling jobs: predictive analysis of job completion time, an intuitive user interface, complete control over job execution through a dashboard and links with system management, and the ability to expand processing capacity as needed. According to our study: Users of enterprise workload automation solutions are reaping benefits that nonusers underestimate. Of the IT decision-makers who currently use an enterprise job scheduler, 61% claimed to receive better control of all

8 6 FIGURE 7 Use Of Enterprisewide Job Scheduler Drives Benefits Including Better Control And Error Reduction What benefits of workload automation do you anticipate realizing or have you realized by using an enterprise job scheduler? (Select all apply) It optimizes server usage and available capacity It provides better control of all batch activities It improves the on-time run of batch jobs It reduces manual interventions and errors It reduces the number of FTEs in charge of batch scheduling Nonusers (N = 101) 34% 44% 50% 47% 46% It provides better control of all batch activities It reduces manual interventions and errors It optimizes server usage and available capacity It reduces the number of FTEs in charge of batch scheduling It improves the on-time run of batch jobs Workload automation users (N = 79) 61% 59% 58% 49% 48% Base: 180 IT decision-makers responsible for workload automation and job scheduling at enterprises within the US, the UK, Germany, and France Implementing workload automation speeds job runs. Seventy-four percent of enterprises that implemented workload automation were better able to transfer critical files in time for critical job runs (see Figure 8). In addition, 72% indicated that workload automation allowed critical jobs to be completed in time for business. Seventy-one percent experienced increased control through a single enterprise view across all batch workflows of all applications. Integration, performance monitoring, and performance analysis are top feature requirements. As companies seek to improve enterprise scheduling coordination and capabilities, they respond to solutions that integrate with system management and reporting. In addition, they expect performance monitoring with automated provisioning of virtual resource allocation and public cloud resources. The future is bright for enterprise workload automation. Projected adoption levels for enterprise workload automation suggest that organizations expect significant value in moving to an enterprise scheduler. According to our study, 70% of enterprises plan to migrate within the next 12 months. The 30% of IT organizations that delay or refrain from adoption will be left wading through the cumbersome issues associated with asynchronous schedulers with ubiquitous yet incompatible demands for efficiency.

9 7 FIGURE 8 Workload Automation Enables Timely Transfer Of Critical Files For Critical Job Runs How significant of an impact does the use of workload automation instead of multiple job scheduling tools have on each of the items below? (1 to 5 scale where 5 = significant impact ) Critical files being transferred in time for critical job runs % 48% 74% Critical jobs completed in time for business 33% 39% 72% Synchronization between job runs requiring excessive manual intervention 31% 72% Single enterprise view across batch workflows of all applications 30% 71% No more scripting 22% 45% 67% No manual work to run jobs on different platforms than the scheduled platform for the job 24% 65% Base: 180 IT decision-makers responsible for workload automation and job scheduling at enterprises within the US, the UK, Germany, and France

10 8 Key Recommendations Enterprise must realize that the tools used to automate IT processes today will extend to automating business processes tomorrow. Enterprises need to shift tool decision-making from tactical to strategic and select vendors that address today's needs while satisfying an architectural vision for the future. Selecting a product in this market requires a clear understanding of the enterprise's strategic evolution in terms of infrastructure, applications, and organization. A list of major requirements must be drafted, prioritized, and compared with features offered by the products available on the market today and in the foreseeable future (12 to 24 months). To successfully implement workload automation, enterprises must: Understand the difference between job scheduling and workload automation. Be sure to review the feature function of the proposed solution to ascertain that it includes the following: integration with monitoring and infrastructure management, integration of secure file transfer and all forms of scheduling into a single point of control, advanced features such as resource provisioning, and an on-time completion guarantee. Make sure that this solution integrates with what is currently installed. Workload automation vendors are used to migrating workflows and jobs from one solution to the next. If you are a packaged application user, be aware that supporting a packaged scheduler may mean working with very different levels of job scheduling integrations, from a simple GUI translator to a complete consolidation of scheduling queues. Because IT monitoring integration is important, make sure that the solution integrates with the monitoring, capacity, and provisioning solutions that you have already acquired. Ask for a proof of concept. You need a proof of concept and references before embarking on a workload automation consolidation; you must make sure that the product does what you need in terms of migration and integration. A proof of concept will dispel any doubt about the effectiveness of integrating with your management products. Understand the vendor road map and its robustness. Although the future is still uncertain, the versatility of the solution is key. Select a vendor whose product road map not only follows the latest workload automation features but also offers the best guarantees that it will be executed as needed.

11 9 Appendix A: Methodology In this study, Forrester conducted an online survey of 180 IT decision-makers responsible for workload automation and job scheduling at enterprises within the US, the UK, Germany, and France. Questions provided to the participants were built to understand their current usage of job schedulers and plans for adoption of an enterprisewide job scheduler. The study was completed in May Appendix B: Demographics FIGURE 9 Respondent Job Position And Level Of Responsibility Which title best describes your position at your organization? Manager 13% C-level executive (e.g., CEO, CMO) 36% What is your level of responsibility when it comes to workload automation or job scheduling at your organization? 71% 87% of respondents at the director level and above 100% IT 24% Director 33% 5% Vice president I influence decisions related to my organization s workload automation strategy I am part of a team making decisions for my organization s workload automation strategy I am the final decision-maker for my organization s workload automation strategy Base: 180 IT decision-makers responsible for workload automation and job scheduling at enterprises within the US, the UK, Germany, and France