Remote Control of VMT

Transcription

1 Remote Control of VMT Final Report Review Contract: PWSRCAC Board Meeting January 23-24, 2014 Hisey and Associates, LLC

2 Overview Scope of Contract Definition of Key Terms Review Process Applicable Regulations and Standards Alyeska documents Employee interviews Other publicly available documents Summary of Findings Recommendations

3 Scope of Contract 1. Identify assets at the VMT that are subject to remote control and those subject to local control. Include an analysis of the interface between the two classes of assets. 2. Analyze command and control design, capability, visibility and effectiveness in managing normal and abnormal operations at the VMT. 3. Review the training requirements and procedures for OCC operators relative to VMTspecific activities. Evaluate conformance to operating procedures and requirements. 4. Assesshow the four key processes at the VMT (Tank Storage, Vessel Loading, Power Generation, and Ballast Water Treatment) are controlled at the VMT and/or the OCC and how operations and system status are coordinated and communicated between the VMT and OCC controllers. 5. Assessthe extent to which moving the OCC from the VMT to the Anchorage location changed the limited control capabilities the OCC has for four VMT processes identified in #4 above. 6. Provide Alyeska with recommendations as appropriate for improving the control of assets at the Valdez Marine Terminal during both routine and abnormal operations. 7. Provide reasonable amount of consultation to third parties regarding results, findings, and recommendations associated with work performed under this agreement.

4 Definition of Key Terms OCC Operations Control Center (Anchorage) SCADA Supervisory Control and Data Acquisition o DCS Distributed Control System o PLC Programmable Logic Controller o HMI Human Machine Interface o UCOS User Configurable Open System (TAPS control system software) Controller Person who operates system from console Alarm Management proper design, analysis, rationalization, implementation, monitoring and improvement of control system alarms

5

6 Operations Control Center (Anchorage)

7 Typical System Review Process Review applicable regulations, standards and industry best-practice Review company policies, procedures, requirements, documents and records Interview knowledgeable employees responsible for implementing procedures, etc. Validate with documented evidence Assess company practices and employee understanding against all requirements Document findings and recommendations

8 Modified Review Process Alyeska provided many of their standards, policies and procedures, but provided few documents to demonstrate conformance and very limited employee interviews Scope extended to consider publicly available Alyeska documents involving control system issues FERC Tariff Litigation State Assessment Review Board (SARB) hearings Evaluate possible VMT control system related issues or similar risks

9 Applicable Regulations and Standards Federal Regulations (49 CFR Part 195) DOT/PHMSA Liquid Pipelines. Same control system for Pipeline and VMT Industry Standard Practices o API RP 1165 SCADA Displays o API RP 1167 SCADA Alarm Management o API RP 1168 Control Room Management o ANSI/ISA 18.2 Alarm Management

10 Alyeska Documents Many documents requested were deemed out of scope by Alyeska and not provided All documents provided by Alyeska were identified as confidential Pipeline references were redacted by Alyeska Many documents (forms) were provided blank Largely represent policies, procedures and forms exist Do not provide evidence that policies and procedures are being followed

11 Employee Interviews Interviews originally planned were subsequently denied Interviewed one automation engineer and one Valdez technician All questions were provided to Alyeska in advance of interviews Individuals were largely unfamiliar with subject matter of questions, deferring responses to others who were not made available OCC Manager responded to one of nine sets of questions posed. Responses were extremely brief. Unable to test/validate conformance to standards

12 Other Publicly Available Documents FERC TAPS Tariff Case - Pipeline Strategic Reconfiguration (prudence challenge) Case also addressed issues with implementation of control system changes made since 2007 Searched public documents re: OCC and control system Assessed potential for similar issues related to VMT 2013 State (tax) Assessment Review Board hearings Project Work Request (PWR) involving critical control system communication building replacement at VMT

13 Summary of Findings Some control system documents do not contain sufficient detail BE-20 (Berth Operators Manual) provides appropriate detail PS-9 Crude Tank Overfill Incident findings are relevant to control of VMT VMT Communications Hub / SCADA Building remains in a failed state Key Performance Indicators (KPI) for Control System Alarm Management are inadequate

14 Summary of Findings (continued) Alyeska did not demonstrate having completed Documentation and Rationalization (D&R) of the VMT control system alarms as required by regulations and company standards Response requirements for a berth vapor control building exhaust fan alarm appear inadequate Status of Alyeska commitment to complete a Reliability Centered Maintenance Review of the Control System is unknown

15 Key Findings and Related Recommendations PS09 Tank Overfill Incident root causes and contributing factors included: Design less than adequate (LTA) Previous Incident Management Action Plans (MAP) and Lessons Learned LTA Situational Awareness LTA [OCC Controllers and field personnel] Safe Operating Committees (SOC) LTA Standards, Policies and Administrative Controls (Procedures) LTA To the extent these systemic issues remain, the same factors could contribute to unnecessary risk at VMT. Resolution of these risks needs to be validated by external risk management organization or regulatory agency.

16 Recommendation Alyeska has stated publicly they are evaluating Strategic Reconfiguration type of changes to the VMT. To the extent proposed configuration changes may include a significant increase in process instrumentation and status under the OCC, PWSRCAC should be aware of not only their design, but the specifics of the implementation process, taking into account the type of problems encountered when making the significant changes during the pipeline SR projects

17 Finding and Recommendation Alyeska has a Project Work Request (PWR Z556) in process to replace a critical communication structure at VMT that houses control system hardware PWR submitted as evidence by Alyeska in 2013 state tax hearings The original 2008 PWR states the structure is in a failed state Alyeska now says the term failed state was in error and will be corrected PWSRCAC should request a review/update on the status of PWR Z556

18 Text images from PWR-Z556

19 VMT Communications Building Houses critical communications infrastructure at site of original Operations Control Center

20 Recommendation Alyeska should establish clear measureable goals or Key Performance Indicators (KPIs) to track and monitor their control system alarms.

21 Finding and Recommendation Alyeska did not demonstrate having completed Documentation and Rationalization (D&R) of the VMT control system alarms as required by regulations and company standards PWSRCAC should request documentation from Alyeska to establish the status of the Master Alarm Database (MADB) and D&R for VMT control system alarms

22 Finding and Recommendation OCC Terminal Controller training is critical to safe operation of VMT Training records for VMT Controllers were not provided for review PWSRCAC should request Alyeska provide a status of VMT Controller training

23 Finding and Recommendation Alyeska committed to the Joint Pipeline Office (JPO) in to complete a Reliability Centered Maintenance (RCM) Review on the TAPS Control System PWSRCAC should ask if Alyeska has completed an RCM analysis of the control system, and if it was updated after the relocation of the OCC from Valdez to Anchorage