THE GUIDE SESSION 5 Operations, Administration & Maintenance Considerations for a A Submarine Cable Network

Transcription

1 THE GUIDE SESSION 5 Operations, Administration & Maintenance Considerations for a A Submarine Cable Network Presenter: Peter Barletto (University of Washington)

2 Presenter Profile Pete Barletto continues supporting the University of Washington as a consultant having retired from his post as Director and Chief Operating Officer. While working for Tyco Telecommunications, Barletto was involved in the planning, implementation and operation of the former Tyco Global Network. More recently, he was Vice President of Network Services and Operations for VSNL and was responsible for Operations and Maintenance planning. Presenter Name: Pete Barletto Title: Consulting Director barletto@uw.edu Slide 2

3 Purpose: to provide a high-level understanding of the value of expert operations to the performance and costs involved in Submarine Cable Networks. Outline for this session: Operations, Administration & Maintenance (OA&M) OA&M Roles during: Phase 1 Planning Phase 2 Procurement & Finance Phase 3 Construction Phase 4 Acceptance & Operations Key Elements of Each Section: The Operational value add Lessons Learned The GUIDE: Network Operations Slide 3

4 The GUIDE: Network Operations First some definitions: What are: Operations Administration Maintenance? Who are: the Operators? Slide 4

5 NETWORK OPERATIONS: OA&M is: OPERATIONS ARE: The daily Network management & surveillance including, as may be defined in the C&MA or other agreements; Backhaul Fronthaul Wet Plant Dry Plant Restoration Security and Safety ADMINISTRATION FUNCTIONS ARE: Resource management people, equipment, and support. Contract and vendor management Performance and Financial Management & reporting Configuration and Capacity Management Slide 5

6 NETWORK OPERATIONS OA&M is: Maintenance functions include: Maintenance planning repair & restoration plans and agreements Repair planning and implementation Spares and consumable inventory, storage, and procurement Documentation and drawing updates Event analysis and corrective action recommendations Slide 6

7 NETWORK OPERATIONS The Operators are: There are many variations on how cable owners procure or assign operations responsibilities. Some common items that are typically outsourced to third party providers might include: Wet repair and storage services Backhaul and or Fronthaul services Remote hands maintenance in some cases Historical Consortium arrangement The Assignment & Restoration Sub-Committee. Budget & Finance Construction Slide 7

8 NETWORK OPERATIONS The Operators are: One or more landing parties may be assigned Maintenance Authority responsibility There may also be a party designated as Restoration Liaison Office (RLO) Variations on Operational responsibility include: Single Party overall responsibility Third party with overall responsibility Supplier provided services Overall Network Operations Center (NOC) responsibility is typically assigned to one of the landing parties. Integration into existing NOC facilities is common. Slide 8

9 PHASE 1: OA&M ASPECTS IN PLANNING Goal: Developing a broad OA&M strategy (who and how) and budget as part of project feasibility Slide 9

10 GUIDE Session 5- Outline of OA&M Activities in Each Phase Phase 1 (Planning): System Design& Configuration Impact Landing Party Selection Impact Business Plan input Phase 2 (Project Development): Influence OA&M Issues in Ownership Agreements Influence Technical Specs in Supply Contract Phase 3 (Construction): Participate in Key Construction Reviews Key Station Personnel on-site starting at Construction Phase NOC Development Operations Procedures Development Staff Recruitment & Training Establish 3 rd Party Service Contracts Phase 4 (Owning & Operating): Routine Maintenance Fault Management Network Provisioning Customer Support Restoration Onward Connectivity Slide 10

11 Planning: OA&M Potential Input System Design and Cost Analysis Affects Future Network Performance, Operations, and Maintenance Costs Network configurations OA&M-related risks have long-term impact and warrant consideration Physical as well as political risks may be better known locally Operational experience in specific geographic areas may help to reduce risk Available resources (or not), e.g. maintenance services, can modify cost estimates Technology choices may make an operational cost difference Landing Party Selection Operational relationships and experience can help where choices exist. This begins the operational responsibilities & organizational planning Slide 11

12 Provide Critical OA&M Strategies and Long-term OpEx Forecast Into the Business Plan The strategy serves as initial guidance to Operations Plan The project development timeline determines the timeline for securing critical OA&M resources & developing procedures & tools OA&M is the major element of project OpEx costs over the project lifecycle Network Operations Center: $/yr, where, who provides? NOC POP Planning: OA&M Interest Backhaul Options & Annual costs Cable Station OA&M Services; $/yr, who provides? Wet Maintenance Options And $/yr: Slide 12

13 Planning: Key Lessons Learned Annual Operations Expenses can be significantly affected by local labor availability, regulatory issues and customs. Unavailable, unreliable or insufficient backhaul can add long term costs and performance issues. Local commercial power status and performance could raise operations expenses and system reliability issues. Availability of Marine Maintenance opportunities (ships, depots, etc.). Geological and or political risks. Slide 13

14 PHASE 2: OA&M Influence During Project Development Goal: Assure Final Network Plan Provides Basis for Effective OA&M Slide 14

15 GUIDE Session 5- Outline of OA&M Activities in Each Phase Phase 1 (Planning): System Design& Configuration Impact Landing Party Selection Impact Business Plan input Phase 2 (Project Development): Influence OA&M Issues in Ownership Agreements Influence Technical Specs in Supply Contract Phase 3 (Construction): Participate in Key Construction Reviews Key Station Personnel on-site starting at Construction Phase NOC Development Operations Procedures Development Staff Recruitment & Training Establish 3 rd Party Service Contracts Phase 4 (Owning & Operating): Routine Maintenance Fault Management Network Provisioning Customer Support Restoration Onward Connectivity Slide 15

16 Project Development: OA&M Input & Interest Support OA&M Aspects of Owner Agreements (C&MA, LPA, etc.) Define network maintenance strategy Identification of operations roles and responsibilities (e.g. subcommittee oversight, local landing party role, central coordination role, 3 rd party services) Define key performance metrics and service requirements Agree on budgets (what is reimbursed and what is not) Local knowledge (operations personnel) may be important to making good decisions Develop/Review OA&M aspects of System Supply RFQ/Contract Network Performance Requirements Maintenance Tools (NMS, supervisory system, monitoring, etc.) Documentation & Training Requirements Spares & Long-term Support Requirements Slide 16

17 Project Development: Key Lessons Learned Annual Operations Expenses will be dependent directly on third party operations and, or maintenance agreements and indirectly on the reliability and availability of resources. Annual landing station costs vary significantly and can range from $100K s to more than $2M. They can include: Space leases ($ k s) Taxes & Fees (business, real estate, ROW) (~$1000 s - $2M) Power ($100K 300K) Personnel ($150K - $800K) Miscellaneous third party costs: ($200K 400K) Building maintenance, Security Communications Shipping, postage, travel, training Interconnect Charges, Onward Connectivity costs Slide 17

18 Project Development: Key Lessons Learned Buy versus Build decisions should consider the long term Cost of Ownership versus the short term expedient solution. ($100K s per year for leased bandwidth versus dark fiber + capital) Network Operators combined planning meetings are needed to confirm understanding of roles, responsibilities and commitment to the Maintenance Strategy. Operations Plans and Process development needs to begin and include relevant project milestones for resource availability, training, readiness testing, and acceptance. Inspections and acceptance with a view to long term maintainability should be incorporated. Initial staffing of operations personnel with appropriate skill sets. Operations planning underway for: Training, Safety, Security Slide 18

19 Project Development: Facilities and Equipment Slide 19

20 PHASE 3: OA&M Planning During Construction Goal: Develop or Secure OA&M Facilities, Tools & Procedures, Resources and Services to Support Network Service When Cable System Complete Slide 20

21 GUIDE Session 5- Outline of OA&M Activities in Each Phase Phase 1 (Planning): System Design& Configuration Impact Landing Party Selection Impact Business Plan input Phase 2 (Project Development): Influence OA&M Issues in Ownership Agreements Influence Technical Specs in Supply Contract Phase 3 (Construction): Participate in Key Reviews Key Station Personnel on-site starting at Construction Phase NOC Development Operations Procedures Development Staff Recruitment & Training Establish 3 rd Party Contracts Phase 4 (Owning & Operating): Routine Maintenance Fault Management Network Provisioning Customer Support Restoration Onward Connectivity Slide 21

22 Network Operations Center (NOC): Establishing Critical Control Every cable network needs a central control center NOCs can be simple or sophisticated, but whicheverthey need to support all aspects of customer support and network operations Establishing the facility, tools, procedures, and team to run a NOC needs to be done in advance of service (during construction) Integration into an existing NOC is not simple. Slide 22

23 Constructing: OA&M Input & Interest Staffing Early hiring of key staff is critical Participation in key reviews important to project success Process development initiation with involved staff Supplier and System specific staff training Recruiting and filing out staff requirements (more is better) Landing Party, Station, Fronthaul and Backhaul readiness Contracts in place Operations planning needs to manage readiness of assets, resources and trained personnel Station security, communications, HVAC and safety provisions are required to be available. Slide 23

24 Constructing: OA&M Input & Interest Site Acceptance & Commissioning The Operational role may vary, but at a minimum; New Landing Station Facilities: Station Manager on roll. Building and building systems acceptance Provide a clean and safe environment Witness or perform testing and inspections Document test results Make appropriate recommendations Utilize the learning opportunities afforded In advance of any equipment deliveries. Site testing & acceptance of cable system equipment: Observe and monitor installation Witness and record all testing and inspections Provide recommendations, create & manage punch list Slide 24

25 Constructing: OA&M Responsibility System Installation Oversight Operation support during Wet system installation as Owners Representative The vested interest operations personnel have will insure careful oversight, documentation and recommendations. Process validation and personnel training Landing Location Readiness Station (communications, security, safety, etc ) Fronthaul in place, tested, maintenance agreement. Backhaul in place, tested, maintenance services Onward connectivity in place, tested and maintainable Installation Oversight Witness & document suppliers activities Assist or perform Owners tests Develop and track Landing Location Punch List Slide 25

26 Constructing: Key Lessons Learned Local Knowledge and relationships can aid in the approval processes for permits, licenses, etc... Schedule delays due to late delivery of backhaul, or station readiness are costly. Availability and involvement of Operations staff provides an enhanced level of training above books and classes. Generous early staffing levels pay dividends with early staff losses, infant mortality, higher early fault rates, and process immaturity. Permitting and regulatory approval always requires more effort than expected. Third party support can be valuable in managing the multitude of submissions. Slide 26

27 Constructing: Key Lessons Learned Communications and security must be in place prior to installation. Basic building infrastructure is required and must meet appropriate levels of performance prior to installation start. Planned staging area(s) are needed and plans for receiving equipment shipments are important. Receipt, distribution and storage of Supplier Documentation, such as; Operations Handbooks, equipment manuals, as built drawings, etc needs to be planned and agreed. Critical contacts need to be posted and distributed. (Management chain, local police, fire, environmental, etc ) Slide 27

28 Constructing: Key Lessons Learned Staging areas Early staffing Slide 28

29 PHASE 4: Owning & Operating Goal: Establish and manage the framework (facilities, agreements and procedures) to assure quality services are profitably delivered to network users Slide 29

30 GUIDE Session 5- Outline of OA&M Activities in Each Phase Phase 1 (Planning): System Design& Configuration Impact Landing Party Selection Impact Business Plan input Phase 2 (Project Development): Influence OA&M Issues in Ownership Agreements Influence Technical Specs in Supply Contract Phase 3 (Construction): Participate in Key Construction Reviews NOC Development Operations Procedures Development Staff Recruitment & Training Establish 3 rd Party Service Contracts Phase 4 (Owning & Operating): Routine Maintenance Fault Management Network Provisioning Customer Support Restoration Onward Connectivity Slide 30

31 Operations, Administration, Maintenance O&M Contracts and Maintenance Plans Establish interfaces with the Network Operations Center Plan, Manage and report on Network and Operational Readiness Implement and manage Backhaul agreements Plan and procure onward capacity requirements Owning & Operating: OA&M Operations Responsibilities Implement capacity provisioning and planning processes Notify customers of and implement support services and procedures including tracking, reporting and analysis. Slide 31

32 These include the day to day routine operational functions of the operations staff. Operations: Network Surveillance Trouble Management Customer interface Coordination of Provisioning Restoration implementation as required Administration Capacity provisioning process and control Resource planning and management Training administration Contract and vendor management Plan and procure onward capacity Performance and Financial reporting Safety, Quality and Security Management & Audits Owning & Operating: Operations, Administration and Maintenance Slide 32

33 Owning & Operating: Maintenance Maintenance planning repair & restoration plans Warranty Management Repair planning and implementation Spares and consumable inventory, storage, and procurement Documentation and drawing updates Event analysis and corrective action recommendations Technical Support Slide 33

34 Owning & Operating: Network Operations Center Establish or interface with the Network Operations Center The NOC responsible party will often have an existing physical NOC facility with some level of 24/7/365 staffing. As a new system or systems are added: Staffing and or training for the new technology, procedures and and systems needs to be implemented and tested. Interface arrangements (contact lists, media, methods) need to be agreed and implemented with all of the involved parties including operations staffs, technical support groups, customers, owners representatives, maintenance providers, etc Operations Support Systems NMS, EMS, should be given specific focus for interconnection, user access in all necessary locations, and of course training. Slide 34

35 Owning & Operating: Readiness Tests The purpose of the Readiness tests are to demonstrate the parties capabilities to operate and maintain the System to the level required and defined by the owners and measured in accordance with agreed metrics and Direct Measures of Quality. Readiness tests follow a prescribed plan and process and should include: Site and Local testing and validation. I.e. Check lists before initiating tests to confirm prerequisites are in place, including: Trained personnel Communications Systems access Documentation Access to equipment Overall Network Readiness testing follows with each of the landing parties with the designated NOC party in the lead. Slide 35

36 Direct Measures of Quality (DMOQ) the two common types of DMOQ s are those related to Cycle time and Performance. Common Cycle Time DMOQ s can include: Mean time to respond (e.g. 2 4 hour to respond reported trouble) Mean time to repair (hours for Dry, days for Wet) Mean time to provision days or weeks Common Performance DMOQ s can include: Transmission performance, e.g. Bit Error Rate, Packet Loss, Q Level Availability as a percentage of time (i.e %) Owning & Operating: Readiness Tests DMOQ s can also be applied to critical support items AC power, HVAC, Security alarm reporting etc Slide 36

37 Owning & Operating: Readiness Tests DMOQs are commonly used within third party agreements as motivation where failure to meet the agreed levels can result in financial credits or rebates. In extreme (and rare) cases repeated failure to meet basic DMOQ s can establish breach of the agreement. Another metric in common use is the Service Level Agreement or SLA, which also sets forth contractual performance levels and penalties. SLA s are often found in bandwidth, real estate leases and long term Indefeasible Right to Use (IRU) agreements. Slide 37

38 When it Fails Shunt Fault Slide 38

39 NETWORK OPERATIONS: Cable Fault Scenario: Fault occurs Alarms, Customer Calls, Quantify, Qualify fault? Wet Fault initial Sectionalization Notify Wet Maintenance Provider Transmit Preliminary Notification Implement restorations Confirm auto restore Slide 39

40 Refining Fault Location Shunt Fault Refine Fault location e.g. ~2km east of Repeater #02 -Supplier provide tools -COTDR -Voltage/Current versus baseline Slide 40

41 Specific Repair Plan Develop repair specific plan with Wet Maintenance Provider Method of repair What spares, fiber types, segment particulars, etc Agree repair plan, schedule, Owner Representative Update Owners Re-schedule other activities. Assign and schedule additional personnel for repair period. Slide 41

42 Coordinate Repair Process Power Safety process. Station & NOC for repair support Review Daily Reports Provide regular Customer updates Oversee and record repair test results. Document new baselines data. Slide 42

43 Complete Repair Restore normal service Notify Customers, Confirm restorations are normalized Schedule normalization of non-autonomous Schedule, conduct and report event analysis Arrange spares and consumable replenishment Confirm all documentation, drawings, SLDs, baseline data are complete. Then take a break Slide 43

44 SUMMARY Key Lessons Learned Early involvement by the personnel and parties with the long term responsibility for OA&M is a key success factor. Technician and Engineer level personnel require six months or more of training and mostly onsite involvement, even with telecom s backgrounds. More than just the specific technology but especially the operations processes and system specific methods and interfaces. Early operational excellence will support Sales and Marketing, and the opposite will be extremely difficult and costly to recover from. Slide 44

45 Operations are the face of your Cable Network and by it s performance will win or lose you market share. Operations are the long term caretaker of your very valuable asset. Operations costs, OpEx, need to be carefully balanced and well understood by the management team(s). Team work among the NOC, Landing Parties and other Operations Team members is a key success factor. SUMMARY: Slide 45

46 QUESTIONS