White Paper Veritas Configuration Manager by Symantec Removing the Risks of Change Management and Impact to Application Availability By Brian Babineau Analyst Intelligent Information Management February 2007
Table of Contents Table of Contents... 1 Introduction... 2 Change Management Software: The Need for Scale, Automation and Risk Mitigation... 3 Evolution of Change Management Solutions... 3 Automating Change Management... 3 Improving Application Availability with an Investment in Enterprise Change Management Solutions... 4 Expediting Change Management with Veritas Configuration Manager... 6 Veritas Configuration Manager Overiew... 6 Veritas Configuration Improves Application Availability for Customers... 7 A Key Component of Server Management... 7 Conclusion... 8 Page 1
Introduction There are several risks IT departments face in order to keep applications and systems online and available. Natural disasters, external security threats such as hackers, internal employees who either mistakenly or maliciously alter configurations and general hardware or software upgrades that do not work as advertised all pose some threat to IT systems, although some are more manageable than others. However, all of these examples, along with other risks, have one thing in common: all cause some level of change to IT systems and software. As infrastructure changes become more frequent, the level of risk to system and application availability also increases, which is driving the need for IT to better manage change and control any adverse impact. One challenge IT departments face involves understanding the current IT system and application configurations. This includes monitoring any changes to these configurations and identifying who made the alterations. Any lack of infrastructure visibility elongates problem resolution when changes are made as IT cannot quickly trace what specific modification was made or identify how to revert back to the previous configuration. Worse, IT may not be able to determine the impact of a configuration change on business applications because they have little insight into when the alteration occurred, who did it, or what applications are dependent on specific systems. When IT upgrades systems, database and application software, the changes are usually made during planned downtime where business applications are taken offline, thereby mitigating the risk that the modifications could cause significant business disruption. Planned downtime occurs because IT cannot accurately model test configurations in line with production environments, making it difficult to determine the potential effect of the modifications. Once completed, validating configuration changes in production environments and ensuring consistency with the test environment also requires additional time and effort. Lastly, without configuration monitoring, IT may make changes to production environments but fail to apply these upgrades to the disaster recovery systems. If a disaster occurs, application failover and other business continuity plans may not work properly. In order to gain control and provide a baseline inventory of IT systems and software, organizations began maintaining spreadsheets of all servers, operating system versions, software licenses and other assets. This concept of tracking IT assets proved that it could help manage expenses and facilitate some change management processes, but it required significant time and resources to maintain these documents. Many solution providers introduced Configuration Management Database (CMDB) software that provided a central repository where configuration information could be collected and stored. With all the information relating to servers, operating systems and other parts of an application infrastructure centralized, IT is able to track its assets and gain a baseline understanding of what systems and software are deployed. However, IT needs solutions that can automate the change management process by populating the CMDB, tracking configuration changes, identifying who made the changes, and reporting anomalies between existing configurations and pre-determined standard configurations. Veritas Configuration Manager by Symantec (VCM), an example of a next generation change management solution, incorporates several features that assist customers by automatically collecting information on application and system infrastructure dependencies, tracking changes made to the infrastructure, and notifying the appropriate individuals when certain elements do not comply with internal standards. As a result of the innovations made in configuration tracking and change management solutions, organizations can reduce the potential risks caused by modifications while improving application availability during alterations. Page 2
Change Management Software: The Need for Scale, Automation and Risk Mitigation Evolution of Change Management Solutions During the later part of the twentieth century, IT departments invested millions of dollars and resources in bringing new applications, often electronic commerce or other online initiatives, online. As more and more hardware and software solutions were deployed, IT hired business analysts to maintain large spreadsheets that kept an inventory of systems, software versions, maintenance plans and even system serial numbers and software license keys. These spreadsheets evolved into more than just IT asset management documents. Eventually, IT attempted to map all related systems and software that supported a particular group. Imagine a workbook where one spreadsheet was the master inventory list and other sheets were used to track assets grouped by application, location or business unit. This rudimentary IT inventory process certainly helped monitor infrastructure elements and slightly improved configuration change management operations during upgrades and other modifications as there was one place to observe which systems were connected to certain applications. Unfortunately, the spreadsheets required constant updating and any input errors could prove costly in the future. Every time a change was made, IT had to update the workbook and, of course, there were always multiple copies of the spreadsheets in various locations, making it nearly impossible to know which file represented the latest version. To centralize and automate configuration management as well as track IT asset inventory, several information management software providers developed CMDB solutions. These solutions provide a centralized location where configuration information can be maintained. A CMDB repository resolves the issue of having multiple versions of spreadsheets that contain configuration data as IT now has one central location to update changes. IT can also design processes and roles around a CMDB solution, defining who has responsibility for upkeep and how often updates are made to the repository. CMDB technology provided IT with a better understanding of configurations and an ability to maintain configuration information more easily than previous methods. Despite the evolution of centralized configuration information, the risks associated with ongoing changes to IT infrastructure still exist. When configuration changes are unintentionally made and disrupt application availability, it may take several hours to identify which systems or software were altered. Currently, many change management processes are supported by manual tasks where configuration information is manually gathered, and in some cases entered into a CMDB, making it difficult to track any changes. System logs are used to collect information on who had access to systems when IT is attempting to identify who made hardware or software modifications. Logs are also screened to troubleshoot what changes were made and the impact of these changes. The problem is that IT may need to correlate log files from applications, databases and middleware in the hopes of determining what happened. Lengthy problem identification and resolution may keep critical business applications down for extended periods of time. During changes such as upgrades, IT can avoid any downtime by understanding what applications will be impacted, providing IT with the appropriate insight to plan ahead and minimize the disruption caused by the changes. The risk to application availability associated with change created a need for automated, change management software that went beyond centralizing configuration data. Automating Change Management Change management software can help IT automate the collection, analysis, and reporting of configuration information as well as track attributes associated with configuration changes. Information such as which user makes configuration modifications, what applications are impacted and when certain changes are made can help IT plan for upgrades and mitigate application downtime resulting from unauthorized and accidental alterations. Page 3
Customers should not have to manually enter and maintain configuration information a task that is nearly impossible in many large IT shops due to the number of devices and the high pace of change that occurs in these systems. Change management software can automatically discover several infrastructure elements and associated attributes including server memory, operating system level, file systems, IP addresses, application versions and actively running application services. This insight is needed to make well-thoughtout configuration changes, keeping applications online and available to business users. Enterprises run several thousand servers, support many operating systems and create complex, multitiered application configurations. Each of these elements contains valuable information that change management software can analyze to provide unique insight into configuration dependencies. For example, a three tier web application may be supported by a specific version of middleware. Any changes or recommended upgrades to the middleware tier could impact application availability. A dependency map is critical for IT to understand what infrastructure elements support certain applications that run the business. As configurations are modified, change management software saves the original configuration and tracks all changes in real-time. Enterprise organizations are constantly modifying system configurations as operating system patches are issued, new servers are deployed and applications are upgraded. By capturing changes in real-time, IT staff can be notified that a modification has been made, giving them a chance to minimize the potential disruption and confusion often caused when a system is altered and no one knows what was done, who did it or when the change was made. Also, an enterprise change management solution should permit IT to establish standard configuration parameters and alert IT when certain infrastructure elements fail to meet the predetermined requirements. For example, IT may want all servers supporting the Customer Relationship Management application to run a certain level of an operating system to prevent against security vulnerabilities. Due to application performance concerns, IT may add a new server to balance workload across the CRM application, but the new server does not have the appropriate operating system patch installed on it. As soon as the server is added and compared against the standard configuration requirements, IT can be notified and immediately patch the new server to bring it inline with corporate standards. This capability ensures that there is no risk to the supporting elements of the critical business applications and is not feasible if IT cannot establish standard configurations for certain elements within an application group. The risk posed by configuration changes led to the evolution of solutions from spreadsheets to automated change management software. Along the way, IT has been able to eliminate manual processes that kept applications offline as a result of modifications disrupting supporting systems. With next generation change management software, IT is able to resolve problems associated with configuration changes, translating to increased application availability. Improving Application Availability with an Investment in Enterprise Change Management Solutions There are several benefits for enterprises interested in leveraging the information collected, analyzed and reported on by change management software. Many organizations deploy systems and servers at a second location for disaster recovery purposes. In some cases, these systems are not upgraded in line with those located in the primary data center. IT may make changes to the primary application environment, but fail to perpetuate the modifications to the disaster recovery site. If for some reason there is a disruption to the primary application infrastructure and the infrastructure is designed to fail over to the remote location, the organization runs the risk that the application may not be compatible with the secondary systems. The remote system s configuration may not support the latest version of the application or there could be other compatibility issues that may prevent the failover from working properly. Page 4
Organizations that implement disaster recovery solutions should monitor and update remote system configurations as well as test failover operability periodically to ensure that the entire process and systems work as the business expects applications to be available given the costs associated with any technology investment. Change management solutions also enable IT to ensure that development deployments are exactly the same as production environments. In situations that are similar to the aforementioned disaster recovery discussion, it is easy to believe that testing is being performed on the latest application or system upgrades of the same configuration that is currently used in production. Problems may arise when the upgrade or other changes are applied to the production environment and do not work as anticipated because the testing occurred against a different configuration. As a result, the changes may lead to extended periods of application downtime beyond what is already planned. In some cases, IT plans for some downtime, but may not be able to plan for the impact of changes that were inaccurately tested against a configuration that is not in synch with the production environment. The need for planned downtime due to changes made to elements of the application infrastructure can be mitigated. For example, a customer operating a three tier application with web, middleware, and database servers may want to switch middleware servers from a UNIX based operating system to a version of Linux in order to save costs. Before making the change, IT can look at change management software reports to see all of servers that make up the three tier application infrastructure, create a similar test environment to determine if the new Linux servers can support the middleware software and then make the modification to the production environment. If for some reason the change does not work, IT can examine the previous configuration and reconnect the old middleware servers just as they were before the alteration. With change management software, IT can double check test environments ensuring configurations align with those deployed in production. As a result, modifications, upgrades and other development activities can be fully checked and scrutinized prior to production deployment. Once completed, change management software can also be used to validate that the modifications were implemented into production as authorized. This use-case for change management software can prevent significant disruption to applications during normal upgrades and other alterations to production systems, and speed the acceptance of changes into production. Organizations may choose to utilize change management software to facilitate compliance with information privacy regulations. When changes to certain systems are made, IT can easily determine if the alterations may increase system vulnerability, thereby increasing the risk of a security breach. In addition, as discussed above, an organization may establish internal standard configurations to assist with compliance. By developing audit trail reports on historical changes and identifying who performed the modifications, organizations can constantly monitor any events that lead to a security breach. Organizations can develop standard configurations and constantly compare existing configurations against them. Reports can be generated to highlight periods when certain configurations are out of compliance with the standard, allowing IT to take action quickly. Complying with internally developed standards can also enable IT to monitor and maintain application service levels, a growing best practice for several organizations that utilize ITIL to help improve IT service management efficiencies. The biggest benefit of maintaining up-to-date development environments is mitigating the impact of change on critical business applications and the associated supporting infrastructure in order to keep applications online and available to business users. Change management software can help IT plan for upcoming modifications. However, in some cases, IT may not be aware of changes that occur. Employees may accidentally modify configuration files within applications or middleware software, server operating system patches may be deployed without testing or there could be hackers trying to access systems and alter permissions in an attempt to steal information. Whatever the case may be, whether it is accidental or intentional, if IT does not know change is coming, the impact to applications can range from minor disruption to extensive downtime. Change management software, because it monitors all changes in realtime, can generate alerts when any modifications are made. IT can compare the new configurations with Page 5
previous ones, quickly identifying what was altered and determining which applications may be impacted. IT can also easily discover who made the changes, allowing them to judge whether they were intentional or accidental. Most importantly, IT can respond swiftly, resolving any problems that may arise. Expediting problem resolution due to planned or unplanned changes keeps applications available as the impact and issues resulting from the change are minimized. Expediting Change Management with Veritas Configuration Manager Veritas Configuration Manager Overiew As organizations look to benefit from change management software, the feature set and scalability of Veritas Configuration Manager (VCM) can help customers automate change management processes and control the impact of configuration changes on applications. There are several features that allow VCM to scale across an enterprise. First, VCM provides a central location to aggregate information from up to 15,000 servers. When collecting information, VCM identifies commercial and custom applications and then gathers data about an application, including the version and active processes. From the physical server, VCM collects the location, operation system, operating system version and patch level, file system, memory, CPU profile and network connectivity. All of this information can be automatically detected and collected from applications, middleware, web, database servers and associated software no manual discovery or data input is required. Once all of the information is gathered and analyzed, VCM dynamically creates dependency maps between running applications and supporting hardware and software. Figure One depicts a screenshot of the dependency mapping within VCM. Customers can group servers by application, business unit, or data center location depending on how service levels are measured as well as how infrastructure elements are deployed. This feature allows IT to track, audit and report on all configurations and configuration changes by specific group, providing application owners with information regarding the infrastructure elements that are most meaningful to them.. Figure One: A Screen Capture from VCM s Dependency Mapping Capability Page 6
Customers can establish gold configurations and compare what exists against the standard. By making these comparisons, IT can constantly measure configurations to ensure consistency across the environment and drive compliance with corporate and regulatory standards. As changes are made to servers, software and other application infrastructure elements, VCM tracks all the changes that are made, when they are made, and who made them - all in real-time. IT can view a history of configurations in case a problem occurs and necessitates a referral to a historical deployment to resolve the issue. Alternative technologies capture changes during periodic scanning of the IT infrastructure elements, which can help alert IT that certain modifications were made. However, if disruptive or noncompliant changes are made outside of the periodic scanning, IT will only know after the alternative product s next scheduled scan. Veritas Configuration Improves Application Availability for Customers Many IT departments spend significant amounts of time trying to diagnose a problem after an upgrade or system configuration change, and in some cases, applications are offline during these exercises. VCM can help identify all changes made to a configuration, which could be the potential source of the problem, and then provide IT with the last known good configuration so the changes can quickly be rolled back or updated. Faster problem resolution, such as reverting back to working configurations, mitigates the risk of conducting large scale changes. VCM customers can utilize the solution to compare existing configurations with historical changes in order to quickly identify and resolve problems, reducing the risk that applications are unavailable for several hours or worse, days. IT can also use VCM to monitor development and disaster recovery implementations, helping IT complete planned changes as quickly as possible and mitigating the impact of unplanned disruptions to configurations. By tracking all changes in real-time, VCM can generate alerts and reports if a configuration fails to comply with a gold standard. With the Datasync API, VCM can propagate configuration information and relationships to a higher-level IT framework system such as HP OpenView and IBM Tivoli. Real-time notification of changes made to an application infrastructure can prevent application downtime as IT is immediately made aware of a configuration infraction when compared to the gold standard. In some cases, reporting capabilities can help IT identify particular configuration changes that increase susceptibility to system security breaches as well as identify internal employees that may be maliciously attempting to hack into applications. In addition to faster problem resolution, VCM customers can utilize the solution to manage server and software inventory. The benefits of knowing exactly how many servers are running specific software manifests themselves in not overpaying for licenses and maintenance renewals. The comprehensive coverage of VCM means that IT does not need multiple solutions to manage an inventory of applications, server hardware, operating systems, database software and other infrastructure elements. As customers become more familiar with VCM, they can utilize more of the configuration and change management features to garner additional benefits such as improving application availability during upgrades or disaster recovery planning that go beyond IT inventory control. A Key Component of Server Management Change management is one challenge that IT departments must control and VCM facilitates the automation of several steps of the configuration change management process. With VCM, IT can develop standard configurations and constantly monitor production, development and disaster recovery environments, ensuring consistency. If certain configurations fail to meet the standard, IT may need to deploy operating system patches, upgrade application software, or introduce high availability deployments such as clustering to help keep applications online and available. With the appropriate solution, IT can automate patch deployments and utilize VCM to test upgrades and software updates in a development environment that is in synch with production. Page 7
Clustered servers for high availability may be part of a standard configuration for running mission critical applications and can also be tracked by VCM. Inconsistencies in clustered nodes can prevent successful failover from occurring, but VCM can enable notification of changes to these mission critical environments, helping to guarantee the performance of high availability clustered solutions. After configurations are up to date, IT may want to optimize server resources by moving applications to better performing hardware. The movement of applications can also assist in business continuity planning as any server failure can be mitigated if an application can be automatically moved to another device with the appropriate resources. Customers that standardize, optimize and increase availability during normal standard server management activities can benefit from VCM as well as other solutions within the Veritas Server Foundation family. VCM delivers the benefits of tracking configurations, which is extremely important and can help IT make decisions when altering configurations with patches and other updates. In addition, VCM can assist in the locating of servers that may not be implemented with the appropriate failover software. To mitigate risk of application downtime caused by hardware failure, IT can deploy application resource optimization and clustering software, both of which are part of the Veritas Server Foundation family. Customers can work with one vendor to automate server management while keeping applications available as modifications are made. Conclusion Whether it is maintaining a disaster recovery site, deploying new servers to increase application performance or trying to complete database software upgrades, changes in the data center increase the risk of application downtime and pose unique challenges for IT. As configuration changes are planned, it is necessary to identify what applications will be impacted, determine if the new configuration complies with internal security policies, and most importantly, try to mitigate the impact to applications caused by the modifications. To mitigate this risk to IT systems and business applications, it is key that the proper staff members have access to the appropriate configuration information in order to properly test upgrades and patch updates, and to prepare for changes. In some cases, IT simply cannot plan for configuration changes. Employees may make alterations to certain software or upgrade an operating system without notifying data center managers and other IT administrative staff. When unplanned changes are made, IT goes into fire drill mode and attempts to identify what happened and how to fix it. Without the appropriate insight, IT cannot determine what applications may be impacted or the potential risks associated with certain modifications. As configuration change management challenges worsen, innovations in software solutions to automate configuration information collection and analysis were introduced. Change management software enables IT to understand what they have, how it is configured, when changes are made to these configurations and who made the changes. By tracking configuration changes in real-time, IT can easily determine what applications may be impacted by certain alterations, and swiftly resolve any issues that might result from the modifications. Change management software also allows IT to maintain development and disaster recovery deployments that are consistent with production, further mitigating the risk of application downtime caused by configuration changes. Symantec s Veritas Configuration Manager, a change management software solution, scales to support thousands of application, middleware, database and web servers. By collecting and tracking configuration information in real-time, VCM provides customers with the ability to test upgrades within a development configuration that is the same as production. VCM s dependency mapping features allow IT to ascertain what applications will be impacted by planned or unexpected configuration changes. Real-time configuration tracking ensures that IT is immediately aware of any changes that are made and if those Page 8
changes pose risk to application availability, IT can identify the modification as well as who made it and then begin problem resolution. While many organizations are not aware of changes made to application infrastructure because of the manual processes associated with tracking configuration data, VCM customers, on the other hand, can begin to improve other aspects of server management such as optimizing performance or increasing system availability. All of the above examples may require some change, but with VCM, organizations can significantly reduce the risk of application downtime when making these improvements. All trademark names are property of their respective companies. Information contained in this publication has been obtained by sources The (ESG) considers to be reliable but is not warranted by ESG. This publication may contain opinions of ESG, which are subject to change from time to time. This publication is copyrighted by The, Inc. and is intended only for use by Subscribers or by persons who have purchased it directly from ESG. Any reproduction or redistribution of this publication, in whole or in part, whether in hard-copy format, electronically, or otherwise to persons not authorized to receive it, without the express consent of the, Inc., is in violation of U.S. copyright law and will be subject to an action for civil damages and, if applicable, criminal prosecution. Should you have any questions, please contact ESG Client Relations at (508) 482-0188. Page 9