Reduce Production Incidents with Oracle Enterprise Manager 12c and give yourself a break! Roland Evers

About us Roland Evers 34 years old Living in The Netherlands 9 years at Accenture > 5 years Oracle Administrator since EM 10g Tech Area Oracle - Infrastructure & Databases; HW/SW, Infrastructure, Monitoring, Linux Software Engineer Sr. Analyst Oracle Technology Loves: Music, photography, theme parks, a good movie and repairing stuff Oracle Enterprise Manager 12c: Certified Implementation Specialist 2014 Accenture. All rights reserved. 2

Accenture Accenture is a global management consulting, technology services and outsourcing company. Accenture has more than 323,000 people serving clients in more than 120 countries. Combining unparalleled experience, comprehensive capabilities across all industries and business functions, and extensive research on the world s most successful companies, Accenture collaborates with clients to help them become high-performance businesses and governments. The company generated net revenues of US$30.0 billion for the fiscal year ended Aug. 31, 2014. Accenture Positioned as a Leader in 2014 Gartner Magic Quadrant for Oracle Application Implementation Services, Worldwide 2014 Accenture. All rights reserved. 3

Oracle Enterprise Manager 1. Introduction: Business Case & Maturity Levels Concepts, in short on R4 2. Plan: Roadmap & Requirements 3. Oracle Enterprise Manager 12c Monitoring 4. Next Steps: Team & Processes 2014 Accenture. All rights reserved. 4

Executive Summary Situation Design & monitoring in order to provide detection and alerting of service or component failures in business chain is missing Monitoring on application level is extensive but not for all applications on the same level Monitoring on Infrastructure & KPIs is not fully developed Program Objectives Identify monitoring improvement opportunities across infrastructure and its systems; roadmap to enable realization of recommendations Program Approach Create a plan to set up an improved monitoring framework through thorough analysis of current and desired monitoring Include a tooling assessment to find the best quality cost ratio tools Enable 24/7 monitoring management on the newly created monitoring Program Results Phased realization of this plan will ultimately result in a comprehensive monitoring catalog, new monitoring that includes updated or new tooling 2014 Accenture. All rights reserved. 5

Level Zero Level One Level of Maturity Level Two Level Three Level Four Level Five Model for maturity monitoring Monitoring is not existing Monitoring activities are not performed Monitoring is ad-hoc Activities are executed reactive and ad hoc, not clear or defined. Monitoring isrepeatable but intuitive Activities are executed regularly and actively but are executed differently across resources or teams. Monitoring is defined and structural Monitoring activities are defined, activities are executed according to the processes. Monitoring is managed and measurable Activities are being executed actively and pro-actively continuously, according to the defined processes. Execution is monitored and measured. Performance is measured & discussed Monitoring is optimized Activities are being executed pro-actively continuously according to the defined processes. Execution is monitored & measured. Performance is measured and discussed with individuals. Activities are improved based upon measurement, review and evaluation. Non Existing Initial Repeatable Defined Managed Optimized 6 2014 Accenture. All rights reserved. 6

Costs of a Priority 1 Incident Example P1 Financial Impact Cost System Downtime Business non productive, Resources waiting, idle Lost Client Revenue; e.g. in Call Centers Public Image; Publicity, Trust Cost Consultants for Resolving Hourly Rates Time they cannot spent on: Changes, Solving structural Problems, Solving P2 & P3 issues, Innovation Other Factors P1s cannot be planned and are hard to factor into staffing & resourcing planning P1 resolution lowers team productivity: P1 Disrupt Planned work, Changes and solving structural problems P1 Require management focus 7 2014 Accenture. All rights reserved. 7

Oracle Enterprise Manager 1. Introduction: Business Case & Maturity Levels 2. Plan: Roadmap & Requirements 3. Oracle Enterprise Manager 12c Monitoring 4. Best Practices & Next Steps 2014 Accenture. All rights reserved. 8

Event Monitoring Be aware of availability & performance problems 24x7 Specify critical vs. warning thresholds for metrics Various notification methods: email, SNMP trap, OS command Notification rules and schedules for alerts Predefined & user-defined monitoring templates 2014 Accenture. All rights reserved. 9

Levels of monitoring maturity From scattered to analytical monitoring 0 Start Monitoring Project 1 Completed Monitoring Project 2 Monitoring Control Center Analytics Level 0 Scattered Monitoring Level 1 Mature Monitoring Level 2 Centralized Monitoring Level Analytical Monitoring Maturity Level o All kinds of tools o Different teams Maturity Level o One main tool o Different teams o o Maturity Level One main tool One central team Maturity Level o One main tool o One central team o Advanced Analytics Tools o OEM o SCOM o Nagios o Lots of custom scripts Tools o OEM o Less custom scripts o o o o Tools OEM BI Publisher integrated Minimal custom scripts Tools o OEM o BI Publisher integrated o Minimum to no sanity checks 2014 Accenture. All rights reserved. 10

Infra Application Interfaces Components Functionality Monitoring Levels - Technical (L)Unix Solaris Windows Databases Oracle MySQL Web servers WebLogic BRM FMW HCM CRM Finance Web services Batches XML Application Processes Login functionality Severities Availability Reports in time Data health All SW components End-user monitoring and business KPI s are not included in this overview 2014 Accenture. All rights reserved. 11

Oracle Enterprise Manager 1. Introduction: Business Case & Maturity Levels 2. Plan: Roadmap & Requirements 3. Oracle Enterprise Manager 12c Monitoring About System & Services, Administration Groups 4. Best Practices & Next Steps 2014 Accenture. All rights reserved. 12

Automation Challenges no automated monitoring configuration Application deployments with new environments what about monitoring & alerting? New hardware & software: new or re-arrange monitoring set-up Home grown tools (older versions, impact, security, manageability, responsibility, changes ) Various & different applications: require different configurations Similar applications, but different domains Multiple (internal / external) parties and teams with different demands and requirements 2014 Accenture. All rights reserved. 13

Oracle Enterprise Manager One suite for various departments One suite View No separate tools Wide range of Metrics - out of the box Notifications for anyone how they prefer Dashboards to show status of systems and services Root cause with topology Drill down from key business processes down to a single statement / metric Entire Oracle Stack! Combined with the best database management, tuning and analysis Support Innovative Being prepared for: Self service Provisioning The cloud Hybrid cloud management Applications Management Enterprise Ready Framework Cloud Management Alerting Pro-active monitoring with notifications Helping in finding root causes up front (before it is too late ) Middleware Management Database Management Application Quality Management Provisioning and Patching Configuration Management Chargeback and Capacity Planning Exadata and Exalogic Management 2014 Accenture. All rights reserved. 14

Automation Concept > Analyze > Design > Implement Automate monitoring setup with new hardware / software Flexible solution No differences between (application) related servers Role based access Monitoring over different domains Together and possible for other external and internal teams and parties 2014 Accenture. All rights reserved. 15

Automation Administration Groups Concept > Analyze > Design > Implement Automatically place targets into corresponding group: By adding properties to a target (database, host, etc.) Automatic deployment of predefined monitoring checks To all current and future new targets. Saving time, automatic apply collections of templates to similar targets For their specific purpose (HCM, Ordering, Billing, etc.) Incident Rules configured on an Administration group Will automatically apply for any new target in that group. No need to perform ad hoc apply operations or comparing templates settings 2014 Accenture. All rights reserved. 16

Administration Groups Concept > Analyze > Design > Implement Oracle describes Administration groups as: a special type of group used to fully automate application of monitoring and other management settings targets upon joining the group. Concept describes the flexibility in defining your own hierarchy Maximum of four levels (target properties) In general, for most cases: generic target properties (Department & Line of Business) are very logical 2014 Accenture. All rights reserved. 17

Automation - Approach Concept > Analyze > Design > Implement Best Practices: Documentation & white papers Docs.oracle.com Strategies for Scalable, Smarter Monitoring using Oracle Enterprise Manager Cloud Control 12c http://www.oracle.com/technetwork/oem/sys-mgmt/wp-em12c-monitoring-strategies-1564964.pdf Monitoring set-up Administration Groups Templates System & Services Notification Rules Roles Manage incidents Monitoring configurations Fine tuning Incident Rules for notifications Expand Administration Groups 2014 Accenture. All rights reserved. 18

Level Zero Level One Level Two Level Three Level Four Level Five Infra Application Interfaces Components Functionality Automation Concept > Analyze > Design > Implement Analyze: The current situation (Third) parties and external parties Requirements and needs from monitoring levels Break down into all levels OS Databases HW checks Web servers WebLogic BRM FMW HCM CRM Finance SW component s Web services Batches Application Processes Login functionalit y x 2014 Accenture. All rights reserved. 19

Available Criteria OEM 12c Administration Groups Concept > Analyze > Design > Implement Automatic grouping of targets into Administration groups Idea: Administration groups hierarchy: Targets into groups based on following criteria: Lifecycle status Department Line of business Contacts Automatically: Apply Template Collections and set thresholds Include targets in Incident Rules, receive notifications Set privileges for existing Administrators role based access on any group Contact Cost Center Customer Support Identifier Department Lifecycle Status Line of Business Location Target Version Target Type One challenge: Various (external) teams & multiple same Line of Business values within 1 department: we would not be able to use generic names for Departments and or Line of Business. A different approach needed 2014 Accenture. All rights reserved. 20

Administration groups Concept > Analyze > Design > Solution Automated deployment of configurations as targets join groups Target Properties Administration Groups are created based on types of membership criteria levels: Lifecycle Status: Production, Stage, Test, and Development Line of Business: Unique Application identifier Example: Peoplesoft BM, HCM BM, FMW CM, Siebel (BM: Business Market, CM: Consumer Market, etc) Contacts Which party is maintaining the application or application service. Example: Customer s DBA team, Accenture Application Maintenance, other s teams How it works An hierarchy of Administration groups is generated based on these three levels; New targets are added automatically based on the three predefined levels; Existing or new targets that match the definition, will be placed into the corresponding Administration groups. 2014 Accenture. All rights reserved. 22

Administration Groups Targets placed into Administration Groups based upon criteria Lifecycle Status 1 Production 2 Staging 3 Test 4 Development Line of Business 1 BI Applications 2 FMW BR 3 BRM Business 5 WebCenter Contact 1 Accenture Application Maintenance 2 Client DBA team 3 Analytics Team 4 External Teams Admin Group hierarchy Production Staging Test Development BI App X FMW BM BRM Business Siebel CRM Web Center Ext. Team A Ext. Team B Accenture AM Accenture DBA Client DBA Team Analytics Team 2014 Accenture. All rights reserved. 23

Administration groups Targets into groups Example: new targets (CRM & Fusion) Discovered: Properties set for the targets: 3 new WebLogic servers Test, Middleware, DBA Team 2 new Cluster Databases Production, CRM, DBA Team 10 new hosts for CRM Production, CRM, Testers *New targets! 3 WebLogic 2 2 Siebel Servers (Fusion) Middleware Databases Databases Type: Cluster Database Type: WebLogic Server Line of Business: Middleware Line of Business: CRM Lifecycle status: Test Type: Cluster Database Lifecycle status: Contact: Test Team Line of Business: Middleware Production Contact: DBA Team Lifecycle status: Test Contact: DBA Team CRM Production Middleware Production Middleware Test Application Team DBA Team Test Team DBA Team Application & App Servers Databases Servers DB Middleware & App Servers Databases Targets automatically placed into corresponding Administration Groups 2014 Accenture. All rights reserved. 24

Administration groups Multiple templates on Administration groups Example: new targets (CRM & Fusion) Template Collection CRM Prod: WebLogic server template Cluster Databases template Host template Template Collections automatically applied Metric thresholds Metric Extensions (automated deployment onto these targets) Compliance standards & Privileges And more... Template Collection (CRM (CRM) Prod) Thresholds set WebLogic Database Template CRM Production Middleware Production Middleware Test Thresholds set ME set ME set WebLogic application servers Databases Servers DB Application & App Servers Databases Templates automatically applied onto targets in Administration Groups 2014 Accenture. All rights reserved. 25

Administrators & Roles Roles on Administration Groups View and Operator roles for Administrators Administrators with View privileges on Administration Groups Can view all targets in that particular Administration group. Administrators with Operator privileges on Administration Groups Can view all targets in that particular Administration group Can also perform Operator activities when it has Operator rights on the Administration group. Role: View on all targets Role: View sub selection View Consumers CRM_PRD Administration Group CRM Production Role: Operator CRM Admin Overall user view Consumers: (View_Consumers) Role: CRM Viewer View Application & App Servers Databases Administrator AM CRM Consumers: (CRM Administrators viewer (CRM Consumers) Operator 2014 Accenture. All rights reserved. 26

Systems & Services High Level Services: business processes Logical structure of services, systems and business services. From a business processes point of view: High Level services Services delivered / delivery by client One dashboard, high level services One overview your delivery 2014 Accenture. All rights reserved. 27

Systems & Services Monitor the high level services Systems Services Billing - Business Ordering - Consumers Sales Force Automation CRM A logical set of Targets that collectively provides one or more applications or services. Out-of-box systems are provided for Oracle-Packaged applications and database targets. & Service: In an Enterprise, an entity that provides a useful function to its users Sub-Service: Any type of service created using cloud Control. Aggregate Service: Service that consists out of two ore more services called a sub-service. Generic Service: Using this, you can define a service to model and monitor any business process or application 2014 Accenture. All rights reserved. 28

Systems & Services (3/4) Services defined Billing - Business Services Ordering - Consumers Each Service consists out of different Components. For each Service monitoring rules are setup to check the status. If one Component is down, it does not mean the whole service is down or all other service are impacted Sales Force Automation CRM Define: Services Main Business Processes Systems: collection of targets of a service Required metrics (KPI s) Extra metrics required 32 2014 Accenture. All rights reserved. 32

Consumer Business Systems & Services (4/4) Explanation CRM Business Fusion Middleware Billing - Business Ordering - Consumers BRM Billing software Ordering software CRM Consumers Sales Force Automation CRM Call Software Ordering System 33 2014 Accenture. All rights reserved. 33

Alerts & Notifications Event Management An event can have different severities. The ones focused on in the project are: Fatal: Critical: Warning: Corresponding service is no longer available. For example, a monitored target is down (target down event). A fatal severity is the highest level severity and only applies to the Target Availability event type. Immediate action is required in a particular area. The area is either not functional or indicative of imminent problems. Attention is required in a particular area, but the area is still functional. 2x same warning threshold events: 1x critical threshold event 1 New incident 2x same incident Create Incident Create Incident Send e-mail Create Problem Send e-mail 2014 Accenture. All rights reserved. 34

Key Benefits Reducing costs and have fun! Reduce Costs Significantly decreased manual monitoring configuration effort to a minimum Decreased P1 s, cost saving, increased ROI Automated configuration of monitoring and monitoring Improve mean time to resolution Improve Service Levels Proactive monitoring of performance and availability Monitor key performance indicators and metrics Align with Business Demands On going: Make optimization decisions based on KPI s to be defined Create Service Level Agreements and Dashboards 2014 Accenture. All rights reserved. 36

Conclusion Our Experiences Enterprise Manager R4 Flexible, easy to implement and maintain! Many improvements in Administration groups, Services and administration Usability and efficiency for others Administration Groups With the standards levels, there is already a direction of a definition of groups Less may be more: Flexibility! You do not need to include all 4 levels. Take the time to thoroughly evaluate the concept and design Update and include the target properties, during discovery or synchronization. Systems and Services Good concept when in use and described well Take time to go through the documentation If issues may occur: helps to quickly identify what is the actual impact on your main business processes Diagnostics and Packs Reduce time for root cause analysis via diagnostics findings Recommendations to resolution calculated based on performance and configuration data collected across complete stack 2014 Accenture. All rights reserved. 37

Sources Best practices! Oracle documentation Strategies for Scalable, Smarter Monitoring using Oracle Enterprise Manager Cloud Control 12c http://www.oracle.com/technetwork/oem/sys-mgmt/wp-em12c-monitoring-strategies- 1564964.pdf Experiences & differences between current & previous Enterprise Manager versions Investigate time to find your best approach! 2014 Accenture. All rights reserved. 38

Q&A For more information Roland Evers Software Engineer Sr. Analyst Oracle Technology roland.evers@accenture.com linkedin.com/in/rolandevers Erwin Winkelman Software Engineer Sr. Analyst Oracle Technology erwin.winkelman@accenture.com