Hope is Not A Strategy Avoiding 5 Common Network Management Mistakes Matt Gowarty Product Marketing www.netcordia.com
In the News. February 18, 2010 TechCrunch crippled for 4 hours Cause: a unscheduled router change December 17, 2009 Blackberry loses ability to send email Cause: a routine router maintenance caused unplanned result August 3, 2009 - Paypal goes down payment services are lost for over 6 hours! Cause: a "back-end router" complicated by a change that took place August 9, 2009 - Cisco s web site goes down Cause: A human error September 1, 2009 - Gmail goes down worldwide Cause: A change by an admin intended to improve traffic flow
A Few of The Responses I hope it will be much longer than four years before we face a problem like this again. Matt Mullenweg, WordPress Founder on blog outage impacting TechCrunch Something went wrong.these things happen. Crystal Davis, Sprint on the Blackberry outage Is Hope a strategy? Can IT get away with these things happen?
Mistake 1 Manually Dealing with Network Manually Dealing with Network Change
Facts on the Network Infrastructure 60-70% of all network issues are tied to network change Over 70% of day is spent on unplanned work Up to 80% of all IT resources are consumed just to maintain the status quo Over 90% of organizations use significant manual processes for network infrastructure
The IT Capability Gap Importance of network/ risk of downtime Gro owth IT capability gap 0% IT resource/staff 2000 2002 2004 2006 2008 2010
The Growing Pain of Manual Processes The misnomer of A quick & easy manual change Easy change is 3-5 minutes per device Access device, log in, check current setting, change, save, verify, hope nothing breaks Doing the math 4 minutes per device X 2.3 changes per week X 275 devices = 2,530 minutes (over 42 hours) Doesn t include human error risk
Mistake 2 Don t Understand the Difference between Change Management and Managing Change
Change Management Process Request a change Make change/close ticket Review change Schedule ticket Receive approval
Change Management Process Successful? Request a change Review change Within policy? Approval Impact neighbors? Schedule ticket Good change? Change/ close ticket
Mistake 3 Believing Managing Performance and Managing Change Should Be Separate
Performance and Change Are Separate Performance Team Change Team Track up/down status Manage network devices Monitor response time Implement changes and configurations Measure usage and utilization Maintain consistency and standards Verify service level agreements Manage ticket system
Only Meet When Problems Occur Only get together when troubleshooting starts Who caused the problem What happened When did d the cause startt Where is the impact Why is it going on
Mistake 4 Inconsistent/Non-Standard Networks
Key Driver for Compliance Internal Internal best practices gold standard Designed by IT/networking staff Typically defined on paper and taught Generally accepted by IT ISO, ITIL and custom standards d External mandates - governance Required by government or sector Generally disliked by IT Extensive audit and reporting needs PCI, HIPAA, SOX and many more
Common Issue Today for Compliance Define Standards on Paper Wait Until A ditor Report Is Wait Until Auditor Report Is Needed
Five Critical Steps for Compliance Define Standards & Policies Implementation to Device Set Proactive Monitoring & Alerting Violation Remediation & Verification Reporting Up-to-the Minute & Scheduled
Mistake 5 Device Centric Views
Change on Network Neighbors Typically, changes are focused on individual devices Reviews should consider impact on neighboring devices or along the path Challenges occur with unintended consequences on device neighbors Limited view on impact of change increases risk along Limited view on impact of change increases risk along multiple devices
How to Avoid the 5 Common Mistakes
Understanding the Impact of Change Cause & Effect Help user identify hard to find issues See if a change had a positive or negative impact on health Verify if change impacts policy compliance View impact on device neighbors
Enforce Compliance and Standardization Build Consistency Over 140 pre-packaged rules Wizard encoding of complex rule logic Proactive alerts for policy violations Built-in remediation Live and historical Live and historical status, trends and reports
Improve Staff Efficiency and Productivity Empower Staff Automate data collection & analysis Reduce manual time and effort Become proactive Remediation options Multi-user roles Vi b d Views based on individual needs
Meeting New Technology Requirements Build Foundation Understand impact of virtualization, cloud computing, data center consolidation Visibility ibilit into dynamic world Build consistency/ standardization Scalability/ distributed ib t d networks
Don t Fall Into the Same Traps as Others Manage Change Take control of #1 cause of network outages Understand impact of change Use automation to achieve goals Control access Automate policy management
Hope is Not A Strategy Avoiding 5 Common Network Management Mistakes Matt Gowarty Product Marketing www.netcordia.com