BGE s Incident Command System Logistics of Disaster Management

Transcription

1 BGE s Incident Command System Logistics of Disaster Management June 19, 2013 Copyright 2013 by BGE. Published and used by INCOSE and affiliated societies with permission

2 Lecture Agenda Company overview Managing events Incident Command System Example Hurricane Irene Continuous improvement Questions 2

3 Exelon Utilities 1.2 million electric customers 650,000 gas customers Service Territory: 2,300 square miles Member MAMA and SEE 1.6 million electric customers 494,000 gas customers Service Territory: 2,100 square miles Member MAMA and SEE Exelon utilities serves more than 6.6 million electric and 1.2 million gas customers. 3.8 million electric customers Service Territory: 11,300 square miles Member Great Lakes and Mid West 3

4 Life Cycle of an Electric Trouble Job 4

5 OMS Job Prediction OMS Job Created Outage Call OMS Shows Customers Out Substation Line Fuse Line Fuse Line Fuse Transformer Transformer Transformer House Service House Service House Service House Service

6 Job Management by Organization Call Center Receives Call Checks for existing trouble in CC&B for that customer providing feedback OR creates new trouble ticket Receives Trouble Job in Outage Management System Distribution Control Room Analyzes Job Dispatches a First Responder First Responder Receives Job on Mobile Data Terminal and proceeds to Job site Makes safe Restores Service Damage work referred to repair crews Repair Crew Receives Job in MDS Restores remaining service or refers back to 1 st responder to restore 6

7 Restoration Priorities and Job Management 1. Public safety/critical customers receive first priority* * Work closely with MEMA to identify outages affecting public safety and critical infrastructure and coordinate restoration to key traffic signals 2. Next, main power lines and equipment that restore largest number of customers 3. Finally, individual transformers/smaller lines to individual homes/businesses, as well as customers out the longest 1. Transmission Lines Bring bulk power from power plants Connect to several substations 10,000+ customers affected 2. Substations Distribute power to several circuits 6,000+ customers affected 3. Distribution Lines Form power networks connecting communities and industries 1,500+ customers affected Diagram 4. Overhead Lines Serve smaller customer groups or neighborhoods 50+ customers affected 5. Transformers Reduce level to usable voltage for your home or business 1-8 customers affected 6. Line to Your Home or Business 7

8 BGE Storm Preparedness Philosophy Have scalable plans in place to respond to electric system emergencies: Determine what plans are needed and when; Document the plans; Determine when the plans are to be implemented and by who; Train and drill on the plans; Implement the plans when required; Review what was done, make adjustments, then re-train and drill. Since plans are scalable. All employees are assigned a storm response role for all event levels. 8

9 Event Preparations Storm Training Table Top Exercise Actual Weather Events & Critiques Lessons Learned Severe Impact Storm Training Severe Impact Storm Drills Annual Storm Preparedness Cycle consists of multiple training sessions, table top drills, comprehensive drills, critiques and feedback processes to facilitate continuous improvement. 9

10 BGE Version of NIMS Incident Command System ICS was first developed after a series of wildfires in California in 1970 After 9/11, the federal government focused on developing and mandating that emergency responders follow a national allhazards approach to large-scale incidents DHS issued guidelines for two systems: the National Incident Management System (NIMS), incorporating ICS, and the National Response Plan (NRP), which became the National Response Framework ICS is a standardized, on-scene, all-hazard incident management concept, which allows responders to adopt an integrated organizational structure to match the complexities and demands of incidents. 10

11 BGE Version of NIMS Incident Command System ICS is activated by an event Five function organizational foundation at BGE Incident Commander: Sets incident objectives, strategies, and priorities. Overall management responsibility. Planning: Coordinates the restoration/response plan at the job level. Analyzes work, then prioritizes and distributes work to the appropriate groups. Operations: Manages and executes the response plan. Manages resources responsible for execution of the plan. Logistics: Provides support, resources, and services required including opening of staging areas, procurement and delivery of vehicles, supplies, materials, food and lodging when required, etc. Finance/Administration: Monitors costs. Manages acquisition and on-boarding of Mutual Assistance Crews. Documents actions taken during the event. Provides weather updates. Incident Commander Operations Section Planning Section Logistics Section Finance and Administration Section 11

12 BGE s ICS Structure 12

13 BGE s ICS Organization is Scalable Wire Down Manhole Fire Severe Storm Event Size and Complexity Incident Commander The initial Incident Commander (IC) will be the senior-most person on the overhead or service crew. The IC will have responsibility for management of the event from the safety, communications, operations, logistics, and planning perspectives. In the event that the incident grows in complexity, incident command will be handed off (via a face-to-face interaction) from the initial IC to the designated IC for the size and scope of the event. Operations Incident Commander Logistics/Planning The IC will be an individual identified prior to the event with the ability to manage personnel and resources and ensure a successful resolution to the event. The entire ICS structure will not need to be activated, but, in addition to the IC, the Operations and Logistics/Planning sections may be activated (logistics and planning may be combined into a single section). The IC will have responsibility for management of the event, including safety, liaising with government agencies and responders, communications, public information, etc. Crisis Information Team Government Affairs Incident Commander Deputy Incident Commander Operations Logistics Planning Safety/Security Legal Finance & Admin The entire ICS organization should be activated prior to the storm or event (assuming that there is warning). An EOC should be activated in order to promote communication between all response personnel. Frequent conference calls should be held that include all section chiefs 13

14 Staying Ahead of Events - Weather Monitoring and Alerts BGE subscribes to two meteorological service providers. Maintain a database of analog storms. Mobilization and response strategies matched to various Weather Alerts. Receive and review detailed forecast summaries 24x7. BGE and service providers communicate Threat and Alert Levels based on BGE defined criteria. Threat Confidence in achieving Alert Level. Alert Matrix of various predicted weather speeds or amounts. Note: Lx in the table above = Lightning 14

15 Experience Based Alert Triggers Figure 20: May - September Weather Alert Level Matrix Alert Level Weather Event Level 1 Level 2 Level 3 Level 4 Sustained Winds mph mph mph >36 mph Wind Gusts mph mph mph > 56 mph Thunderstorms General Slight Risk Moderate Risk High Risk Lightning Strikes Scattered Occasional Frequent Widespread/Frequent Rainfall 1"-1.5" 1.5-2" 2.1"-3" >3" Tropical Cyclone Review Severe Impact Storm Guidelines Figure 21: October - April Weather Alert Level Matrix Alert Level Weather Event Level 1 Level 2 Level 3 Level 4 Sustained Winds mph mph mph >40 mph Wind Gusts mph mph mph > 65 mph Thunderstorms General Slight Risk Moderate Risk High Risk Lightning Strikes Scattered Occasional Frequent Widespread/Frequent Wet Snow 3-6" 6"-9 9"-12 >12" Dry Snow 6"-12 12"-18 18"-24 >24" Ice <.25".25"-.5".5"-.75" >.75" 15

16 Storm Staffing Preparedness Threat Level Recommended Staffing Anticipated Impact 16

17 Establishing Response Levels and Descriptions Scalable Plans Level & Weather Op Con Definition Customers Affected Typical Weather Conditions Estimated OMS Outages per Event Level Description and Key Activities Level 1 Blue Sky Operations Less than 1% Normal weather 40 Distribution system configuration and activity is within normal parameters Current support level is adequate. Level 2 Adverse Weather Heightened Alert Less than 1% or 3,000 12,000 out Normally associated with heavy wind and moisture 80 Weather is on the system (wind, rain, snow etc.) or there is an imminent threat Notice received from Department of Homeland Security of threat Number of jobs and or level of damage being experienced (or anticipated) exceeds staffing levels dedicated, at the time, specifically to service restoration The Shift Supervisor s decision to declare Adverse Weather is driven by multiple factors, including, but not limited to: weather status; # of jobs; # of customers out; and # of crews currently working service restoration Discretionary initiation of: conference calls, activation of EOB-EOC by Incident Commander and possible activation of local/regional incident command BGE crew holdover, scheduling and/or callouts as required and OH construction crews activated Underground and Substation crews may be utilized Hourly Event Paging and notification to CIT/Government Affairs/Regulators (REQUIRED when 10,000 customers are without service for 3 consecutive hours) Level 3 Minor Storm Enhanced Support 1%-2% or 12,000-25,000 customers out Associated with isolated gusty thunderstorms, causing damage to all or parts of BGE s system 175 Widespread severe weather (icing, winds >40mph sustained) or locally extreme weather existing or anticipated Multiple significant events in progress Activities or events occurring or expected to occur on transmission system Possible activation of local/regional incident command Hourly event paging and event report faxing initiated (REQUIRED) Incident Command may be implemented for severe localized system storm damage System monitoring and possible configuration changes to relieve loading or voltage problems 17

18 Establishing Event Levels and Responses A Scalable Plan Level & Weather Op Con Definition Customers Affected Typical Weather Conditions Estimated OMS Outages per Event Level Description and Key Activities Level 4 Minor Storm Comprehensive Support 2%-8% or 25, ,000 customers out Associated with extreme weather systems (heavy ice, winds >50 mph sustained) causing damage to all or parts of BGE s system 500 Widespread extreme weather existing or anticipated All actions from Level 1 through 3 events (as above) BGE Crew holdover, scheduling and/or callouts as needed TSO/ DSO Shift Manager shall submit required form to the DOE when 50,000 customers without service for >than 1 hr Contractor usage should be considered or activated Mutual Assistance from Exelon Utilities considered or activated Mutual Assistance from other utilities considered Whenever restoration is expected to last beyond 24 hours, leadership should consider opening a Regional Command Center (RCC) to either support additional internal resources or external resources. Level 5 Major Storm Emergency Support 8%-20% or between 100,000 and 250,000 customers Normally associated with tropical storms, hurricanes, significant ice accumulation or heavy wet snowfall, resulting in significant damage to BGE s distribution system 1,500 A transmission event has occurred that requires immediate action to prevent a voltage collapse, cascading transmission outages or significant restoration effort. The 100,000 customer limit is based upon a PSC mandate that Maryland utilities report all major storms (10% of electric customers or 100,000 electric customers interrupted, whichever is less). All actions from Level 1 through 4 events (as above) BGE Crew holdover, scheduling and/or callouts as needed Activation of RCC(s) Activation of Load Shedding Plan Level 6 Severe Impact Catastrophe 20% to 100% approximately from 250,000 customers to all Normally associated with the impacts associated with tropical storms, hurricanes, significant ice accumulation or heavy wet snowfall 10,000 These events are characterized by a significant loss of electric service to BGE s customers system-wide, or as a large percentage of a region May cause catastrophic damage to BGE s transmission and distribution systems and the transportation and communication infrastructure Destruction or events causing multiple fatalities along with significant system or facility damage All actions from Level 1 through 5 events (as above) Given such widespread damage, acquisition and use of external crews would be essential. 18

19 Hypothetic Impact and Response Internal Response Staffing Internal staffing working planned work Blue Sky Trouble Additional Trouble (weather) Plans in place to adjust/increase staffing, add support, and alter job management in advance of these types of events to help mitigate the potential of this happening 19

20 Expanding the Response Staff Utilizing Mutual Assistance Most events use available in house resources Mutual assistances is used for extreme events 20

21 Expanding the Response Staff Utilizing Mutual Assistance BGE considers Mutual Assistance in advance of significant events like hurricanes, blizzards, ice storms, or other similar events. Mutual Assistance Groups: Maryland Utilities Group for Mutual Assistance (MUGMA) Mid-Atlantic Mutual Assistance Group (MAMA) Southeastern Electric Exchange (SEE) After an event has started, the need for Mutual Assistance is re-evaluated if it appears that the restoration will take more than hours from the time the event clears the system Additionally, standardize practices are under development to enable crews from the three EU utilities to seamlessly support restoration efforts on one another s systems when needed. 21

22 The large number of off-system crews required for large events necessitates securing a site(s) where these crews can assemble and be managed. Such a site is referred to as a Staging Area. 22

23 BGE Staging Area Locations 23

24 Staffing by Weather Event Type Additional Control Room/IFS Staff Response Organization Staffed At ~75,000 Out Hybrid Response EOC Staff increased by opening RCC. Staging Area may be opened External Crews may be requested and Staging Areas or additional Staging Areas opened External Crews/Mutual Assistance requested with one or more RCCs opened as well as one or more Staging Areas 12,000 Customers Out 25,000 Customers Out Level 2 Adverse Weather Level 3 Minor Storm Level 4 Minor Storm Level 5 Major Storm Level 6 Severe Impact Storm 100,000 Customers Out 100,000 Customers Out 24

25 Storm Monitoring and Preparing Example Hurricane Irene 2011 Sunday, Aug. 21 Monday, Aug. 22 Tuesday, Aug. 23 Wednesday, Aug. 24 Thursday, Aug. 25 Friday, Aug. 26 Saturday, Aug. 27 BGE tracking the storm Weather Conference Calls Mutual Assistance Calls SEE, MAMA, MUGMA Requested 650 crews (linemen and 100 tree personnel) through mutual assistance organization External crews begin arriving External crews continued to arrive Requested additional external crews External crews continued to arrive Staging Areas Regional Command Centers Increased our request to 800 crews External crews continued to arrive Requested additional crews External crews continue to arrive Logistics team began investigating possible staging areas BGE began preparations for Severe Impact Storm organization: Mobilization organization began external staffing process Called for volunteers from outside the organization for temporary public safety response Identified Lockheed Martin as a second staging area additional to BWI-Marshall Completed set up of Piney Orchard Regional Command Center Began set up of BWI- Marshall and Lockheed Martin staging areas Scheduled training for public safety crews Identified crew guides for external crews Completed set up of White Marsh Regional Command Center Storm Leadership finalized mobilization plans and prepared notifications Investigated the possibility of Oregon Ridge as a third staging area Mobilized the storm center, crews, patrollers, public safety stand-by personnel and RCC staff Opened BWI and Lockheed Martin staging areas Hurricane Irene enters BGE service territory at 2:30 PM Communications Proactive communication begins: Press release Proactive communication continued: daily press releases media interviews social media postings Earthquake Proactive communication continued: predictive dialer to all residential customers predictive dialer to special needs customers daily press release media interviews social media postings Proactive communication continued: press release media interviews social media postings videos and photos posted Minor Storm Proactive communication continued: 2 nd predictive dialer to special needs customers IVR updated with pre-storm message press release media interviews social media postings videos and photos posted Proactive communication continued: IVR updated press release media interviews social media postings videos and photos posted 200 Customer Contact Center Representatives mobilized and available 25

26 Hurricane Irene 5 days before entering BGE s territory 1 st formal request for external assistance on Tuesday morning 26

27 Off-System Support for Hurricane Irene Over 2,500 overhead and tree resources from 20 states Home States of External Crews Supporting BGE s Hurricane Irene Restoration Efforts MN MI PA MO IL IN OH KY VA TX OK AR LA MS TN AL GA NC SC FL 27

28 Systematic Approach to Scaling-up Trouble Management Electric Operations Building Emergency Operations Center Electric Operations Building Distribution Control Room Severe Minor Large Typical Small Minor Weather Storm Day damage 12,000 Storm Event > 75,000 Necessary Out 75,000 field resources are moved from planned work to restoration work minor adjustment to field and Control Room staffing Regional Command Center 1 Regional Command Center 2 For an area with severe localized damage Mobile Operations Center dispatched Piney Orchard Service Center (south) Howard Service Center (west) Front Street Service Center (central) Cockeysville Service Center (north) Perry Hall Service Center (east) Staging Area 1 Staging Area 2 Staging Area 3 Staging 4 Staging 5 Staging 6 28

29 Potential Scaling Constraints The extent to which the plan can actually be scaled is limited by a number of constraints : Availability of resources; Impact area; Commute time; Logistical challenges; Hotels Staging sites Infrastructure limitations; Work stations Command Centers PCs Command and control Ratios of off-system crews to company crew guides 29

30 Aftermath of Major Storms 30

31 Continuous Improvement Overview We work to identify opportunities for improvement. We hold critiques and benchmark with other utilities. 31

32 Continuous Improvement - Post Irene Action Plan Post Irene Action Items Due Date Develop standardized pre-scripted employee mobilization communications for severe impact storms. April 1 st, 2012 Develop pre-assigned staffing assignments for severe impact storms. April 1 st, 2012 Standardized daily conference call schedules and protocols. April 1 st, 2012 Modify operational IT systems and/or processes to ensure performance does not diminish during peak usage periods. April 1 st, 2012 Modify logistics plan to separate construction and public safety staging sites April 1 st, 2012 Modify logistics plan to utilize major account reps for local hotel chains to aid booking hotel rooms during severe impact storms. April 1 st, 2012 Develop a process to conduct periodic status calls with local officials during major storms. April 1 st, 2012 Interface with local municipalities to reinforce road closure procedures. June 1 st, 2012 Modify technology and logistics plans to include additional computers/printer/copy at Staging sites. July 1 st, 2012 Participate in PSC ETR working group and update ETR practices as needed. July 1 st, 2012 Update and modify customer communications based on lessons learned from Irene. July 1 st, 2012 Review and modify the Damage Assessment process used for severe impact storms based on the lessons learned from Irene. July 1 st, 2012 Review and modify the Patrol & Public Safety process used for severe impact storms based on the lessons learned from Irene. July 1 st, 2012 Review and modify the Job Packaging process used for severe impact storms based on the lessons learned from Irene. July 1 st, 2012 Modify the process for issuing permits and tags during severe impact storms to reduce bottlenecks experienced during Irene. July 1 st, 2012 Modify the materials tracking processes to facilitate better tracking for reporting after severe impact storms. July 1 st, 2012 Modify the process to ensure that accurate and timely data is being provided from the field during a severe impact storm. July 1 st, 2012 Consolidate Severe Impact Storm Rosters. July 1 st, 2012 Incorporate changes from the Irene Lessons Learned teams into the severe impact storm training. August 1 st,

33 Continuous Improvement - 1st extensive use of Smart Meters at BGE during Hurricane Sandy Historically, BGE staff called customers to verify service status prior to assigning work to a crew. Based on job age No calls performed at night. Smart Meters allow staff to verify electric service to the meter at any time of the day. During Hurricane Sandy, a total of 7,347 smart meters (out of 129,00 installed), mostly in Anne Arundel County, were pinged. A total of 387 outage events were successfully removed from the restoration queue preventing unnecessary truck rolls. 33

34 Questions? 34